Jun 08 14:20:35.231517 np0000002315 systemd[1]: Started devstack@neutron-ovn-maintenance-worker.service - Devstack devstack@neutron-ovn-maintenance-worker.service. Jun 08 14:20:36.641636 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.common.config [-] Logging enabled! Jun 08 14:20:36.642068 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-ovn-maintenance-worker version 28.1.0.dev374 Jun 08 14:20:36.642068 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-ovn-maintenance-worker --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini {{(pid=88443) setup_logging /opt/stack/neutron/neutron/common/config.py:125}} Jun 08 14:20:36.642907 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.server.ovn_maintenance [-] OVN maintenance process starting... Jun 08 14:20:36.642955 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "plugin-directory" by "neutron_lib.plugins.directory._create_plugin_directory" {{(pid=88443) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:511}} Jun 08 14:20:36.643166 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG oslo_concurrency.lockutils [-] Lock "plugin-directory" acquired by "neutron_lib.plugins.directory._create_plugin_directory" :: waited 0.000s {{(pid=88443) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:524}} Jun 08 14:20:36.643166 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG oslo_concurrency.lockutils [-] Lock "plugin-directory" "released" by "neutron_lib.plugins.directory._create_plugin_directory" :: held 0.000s {{(pid=88443) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:543}} Jun 08 14:20:36.643360 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "manager" by "neutron.manager.NeutronManager._create_instance" {{(pid=88443) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:511}} Jun 08 14:20:36.643360 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG oslo_concurrency.lockutils [-] Lock "manager" acquired by "neutron.manager.NeutronManager._create_instance" :: waited 0.000s {{(pid=88443) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:524}} Jun 08 14:20:36.643547 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.manager [-] Loading core plugin: ml2 Jun 08 14:20:37.019904 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.020045 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.020127 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.020699 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.020774 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.020836 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.060332 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.060440 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.060483 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.067118 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.067233 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.067320 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.110009 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.110142 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.110176 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.116980 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.117090 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.117175 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.180348 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent after_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.180348 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent after_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.180348 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: segment precommit_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.180348 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: network precommit_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.180577 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: segment before_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.234589 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network before_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.234723 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent before_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.234800 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network before_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.234862 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.234942 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.235018 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment precommit_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.235072 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment precommit_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.235146 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.235210 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.235305 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network after_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.235346 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network precommit_delete 0, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.235392 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port provisioning_complete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.235463 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.235541 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy after_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.235605 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > agent after_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.235668 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.235668 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet precommit_delete 0, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.235790 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.235790 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.235862 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.235955 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > agent after_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.236075 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.236143 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.236204 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.236536 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: network configured {{(pid=88443) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 08 14:20:37.236614 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: port configured {{(pid=88443) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 08 14:20:37.236685 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: subnet configured {{(pid=88443) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 08 14:20:37.236726 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: subnetpool configured {{(pid=88443) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 08 14:20:37.236788 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: security_group configured {{(pid=88443) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 08 14:20:37.236855 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: security_group_rule configured {{(pid=88443) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 08 14:20:37.237046 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['local', 'flat', 'vlan', 'geneve'] Jun 08 14:20:37.237855 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.drivers.type_flat [-] Allowable flat physical_network names: ['public'] Jun 08 14:20:37.238532 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.drivers.type_local [-] ML2 LocalTypeDriver initialization complete Jun 08 14:20:37.239183 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.drivers.type_vlan [-] Network VLAN ranges: OrderedDict({'public': [(1, 4094)]}) Jun 08 14:20:37.239301 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'geneve', 'local', 'vlan'] Jun 08 14:20:37.239442 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.managers [-] Registered types: dict_keys(['flat', 'geneve', 'local', 'vlan']) Jun 08 14:20:37.239537 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.managers [-] Project network_types: ['geneve'] Jun 08 14:20:37.239646 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.managers [-] Configured extension driver names: ['port_security', 'qos'] Jun 08 14:20:37.240242 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: ['port_security', 'qos'] Jun 08 14:20:37.240364 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.managers [-] Registered extension drivers: ['port_security', 'qos'] Jun 08 14:20:37.240434 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['ovn'] Jun 08 14:20:37.281880 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['ovn'] Jun 08 14:20:37.282006 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['ovn'] Jun 08 14:20:37.282006 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.managers [-] No mechanism drivers provide segment reachability information for agent scheduling. Jun 08 14:20:37.282612 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_response 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.282702 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_response 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.304441 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'flat' Jun 08 14:20:37.304613 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.drivers.type_flat [-] ML2 FlatTypeDriver initialization complete Jun 08 14:20:37.304613 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'geneve' Jun 08 14:20:37.304683 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.drivers.type_tunnel [-] geneve ID ranges: [(1, 65536)] Jun 08 14:20:37.344561 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO dbcounter [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Registered counter for database neutron Jun 08 14:20:37.383404 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG oslo_db.sqlalchemy.engines [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=88443) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:325}} Jun 08 14:20:37.763903 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG dbcounter [-] [88443] Writer thread running {{(pid=88443) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:102}} Jun 08 14:20:37.888594 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.managers [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Initializing driver for type 'local' Jun 08 14:20:37.888722 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.managers [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Initializing driver for type 'vlan' Jun 08 14:20:37.993603 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.drivers.type_vlan [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] VlanTypeDriver initialization complete Jun 08 14:20:37.993717 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.managers [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Initializing extension driver 'port_security' Jun 08 14:20:37.993782 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.extensions.port_security [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] PortSecurityExtensionDriver initialization complete Jun 08 14:20:37.993857 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.managers [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Initializing extension driver 'qos' Jun 08 14:20:37.993951 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.plugins.ml2.extensions.qos [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] QosExtensionDriver initialization complete {{(pid=88443) initialize /opt/stack/neutron/neutron/plugins/ml2/extensions/qos.py:31}} Jun 08 14:20:37.994019 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.managers [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Initializing mechanism driver 'ovn' Jun 08 14:20:37.994083 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Starting OVNMechanismDriver Jun 08 14:20:37.995492 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > process before_spawn 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.995604 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > process after_init 55550000, True {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.995690 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > segment after_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.995769 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > segment after_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.995870 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > segment after_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.995960 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > allowed_address_pair before_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.996038 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > security_group precommit_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.996113 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > security_group after_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.996185 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > security_group after_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.996287 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > security_group precommit_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.996371 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > security_group after_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.996462 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > security_group_rule precommit_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.996575 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > security_group_rule after_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.996651 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > security_group_rule before_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.996734 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > address_group after_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.996833 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > address_group after_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.996927 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > address_group after_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.997078 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > qos_plugin after_init 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.997231 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > trunk_plugin after_init 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.997431 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > logging-plugin after_init 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.997833 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.services.logapi.drivers.ovn.driver [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] OVN logging driver registered Jun 08 14:20:37.997926 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: ._register_pvlan_driver at 0x7f676d636e80> pvlan_plugin before_spawn 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.998008 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: process after_init 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.999240 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.plugins.ml2.plugin [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] neutron.plugins.ml2.plugin.Ml2Plugin method _start_rpc_notifiers called with arguments () {} {{(pid=88443) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} Jun 08 14:20:37.999495 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > port after_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.999600 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > port after_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.999682 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > port after_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.999759 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > subnet after_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.999864 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > subnet after_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:37.999963 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > subnet after_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.000039 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > network after_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.000116 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > network after_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.000186 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > network after_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.000267 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > security_group after_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.000360 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > security_group after_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.000437 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > security_group after_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.000523 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > security_group_rule after_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.000604 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > security_group_rule after_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.000677 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > security_group_rule after_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.000751 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > address_group after_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.000822 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > address_group after_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.000907 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > address_group after_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.000990 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.plugins.ml2.ovo_rpc [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ML2 OVO RPC backend initialized. {{(pid=88443) __init__ /opt/stack/neutron/neutron/plugins/ml2/ovo_rpc.py:117}} Jun 08 14:20:38.001697 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.quota [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded quota_driver: . Jun 08 14:20:38.001794 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.plugin [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Modular L2 Plugin initialization complete Jun 08 14:20:38.001937 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading services supported by the core plugin {{(pid=88443) _load_services_from_core_plugin /opt/stack/neutron/neutron/manager.py:170}} Jun 08 14:20:38.002030 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.plugins.ml2.managers [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Got port-security extension from driver 'port_security' Jun 08 14:20:38.002241 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading service plugins: ['ovn-router', 'auto_allocate', 'tag', 'timestamp', 'network_ip_availability', 'flavors', 'revisions'] {{(pid=88443) _load_service_plugins /opt/stack/neutron/neutron/manager.py:195}} Jun 08 14:20:38.002328 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading Plugin: ovn-router Jun 08 14:20:38.010787 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > router before_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.010910 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: port before_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.010994 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: port after_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.011089 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > port precommit_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.011175 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: port before_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.011263 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: port before_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.011354 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > router precommit_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.011434 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > router precommit_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.011508 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > router precommit_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.011587 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: subnet after_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.011752 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.quota.resource_registry [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Tracking information for resource: router configured {{(pid=88443) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 08 14:20:38.011849 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.quota.resource_registry [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Tracking information for resource: floatingip configured {{(pid=88443) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 08 14:20:38.011932 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.services.ovn_l3.plugin [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Starting OVNL3RouterPlugin Jun 08 14:20:38.012129 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > port_forwarding_plugin after_init 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.012301 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > router before_create 55550100, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.012386 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > router precommit_delete 55550100, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.012463 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > router precommit_create 55550100, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.012540 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > router precommit_update 55550100, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.014810 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.services.provider_configuration [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Service providers = [] {{(pid=88443) parse_service_provider_opt /opt/stack/neutron/neutron/services/provider_configuration.py:170}} Jun 08 14:20:38.014911 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.db.servicetype_db [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Adding provider configuration for service L3_ROUTER_NAT {{(pid=88443) add_provider_configuration /opt/stack/neutron/neutron/db/servicetype_db.py:45}} Jun 08 14:20:38.015841 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > router precommit_create 55549900, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.015954 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > router_interface after_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.016039 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > floatingip after_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.016117 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > floatingip precommit_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.016213 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > floatingip precommit_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.016302 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > floatingip after_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.016380 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > floatingip after_status_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.016455 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > floatingip after_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.016528 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > port_forwarding after_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.016601 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > router_interface after_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.016671 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > router after_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.016748 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > router precommit_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.016832 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > router after_delete 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.016920 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > router after_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.017037 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.services.service_base [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded 'neutron.services.ovn_l3.service_providers.ovn.OvnDriver' provider for service L3_ROUTER_NAT {{(pid=88443) load_drivers /opt/stack/neutron/neutron/services/service_base.py:49}} Jun 08 14:20:38.017129 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: > process after_init 55550001, True {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.017239 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Successfully loaded L3_ROUTER_NAT plugin. Description: L3 Router Service Plugin for basic L3 forwarding using OVN {{(pid=88443) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 08 14:20:38.017327 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading Plugin: auto_allocate Jun 08 14:20:38.018054 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: network precommit_update 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.018139 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron_lib.callbacks.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Subscribe: network precommit_create 55550000, False {{(pid=88443) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.018330 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Successfully loaded auto-allocated-topology plugin. Description: Auto Allocated Topology - aka get me a network. {{(pid=88443) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 08 14:20:38.018399 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading Plugin: tag Jun 08 14:20:38.020465 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Successfully loaded TAG plugin. Description: Tag support {{(pid=88443) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 08 14:20:38.020546 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading Plugin: timestamp Jun 08 14:20:38.021540 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Successfully loaded timestamp plugin. Description: Adds timestamps to Neutron resources with standard attributes {{(pid=88443) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 08 14:20:38.021620 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading Plugin: network_ip_availability Jun 08 14:20:38.022918 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Successfully loaded network-ip-availability plugin. Description: Provides IP availability data for each network and subnet. {{(pid=88443) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 08 14:20:38.022996 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading Plugin: flavors Jun 08 14:20:38.023813 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Successfully loaded FLAVORS plugin. Description: Neutron Flavors and Service Profiles manager plugin {{(pid=88443) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 08 14:20:38.023916 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading Plugin: revisions Jun 08 14:20:38.024729 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.manager [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Successfully loaded revision_plugin plugin. Description: Adds revision numbers to resources. {{(pid=88443) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 08 14:20:38.024836 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG oslo_concurrency.lockutils [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Lock "manager" "released" by "neutron.manager.NeutronManager._create_instance" :: held 1.382s {{(pid=88443) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:543}} Jun 08 14:20:38.025031 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] get_extension_paths = OrderedDict({'/opt/stack/neutron/neutron/extensions': 1}) {{(pid=88443) get_extensions_path /opt/stack/neutron/neutron/api/extensions.py:640}} Jun 08 14:20:38.025108 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Initializing extension manager. Jun 08 14:20:38.025343 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: __init__.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.025424 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: __pycache__ {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.025498 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: aap_reject_multicast.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.025712 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Allowed address pairs reject multicast extension" alias="aap-reject-multicast" description="Reject multicast addresses in Allowed Address Pairs" updated="2026-04-01T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.025923 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension aap-reject-multicast not supported by any of loaded plugins Jun 08 14:20:38.026000 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: address_group.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.026268 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Address group" alias="address-group" description="Support address group" updated="2020-07-15T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.026423 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: address-group Jun 08 14:20:38.026495 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: address_scope.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.026728 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Address scope" alias="address-scope" description="Address scopes extension." updated="2015-07-26T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.026858 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: address-scope Jun 08 14:20:38.026952 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: admin_state_down_before_update.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.027163 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Enforce Router's Admin State Down Before Update Extension" alias="router-admin-state-down-before-update" description="Ensure that the admin state of a router is DOWN (admin_state_up=False) before updating the distributed attribute" updated="2019-07-02T15:56:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.027321 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension router-admin-state-down-before-update not supported by any of loaded plugins Jun 08 14:20:38.027396 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: agent.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.027622 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="agent" alias="agent" description="The agent management extension." updated="2013-02-03T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.027747 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: agent Jun 08 14:20:38.027814 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: agent_resources_synced.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.028014 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Agent's Resource View Synced to Placement" alias="agent-resources-synced" description="Stores success/failure of last sync to Placement" updated="2018-12-19T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.028137 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension agent-resources-synced not supported by any of loaded plugins Jun 08 14:20:38.028204 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: agent_sort_key.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.028408 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Agents Sort Key" alias="agent_sort_key" description="Enabling the keyword sort_key for sorting in agents." updated="2013-02-03T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.028526 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension agent_sort_key not supported by any of loaded plugins Jun 08 14:20:38.028593 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: allowedaddresspairs.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.029152 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Allowed Address Pairs" alias="allowed-address-pairs" description="Provides allowed address pairs" updated="2013-07-23T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.029307 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: allowed-address-pairs Jun 08 14:20:38.029390 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: auto_allocated_topology.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.029690 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Auto Allocated Topology Services" alias="auto-allocated-topology" description="Auto Allocated Topology Services." updated="2016-01-01T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.029835 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: auto-allocated-topology Jun 08 14:20:38.029931 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: availability_zone.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.030297 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Availability Zone" alias="availability_zone" description="The availability zone extension." updated="2015-01-01T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.030441 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: availability_zone Jun 08 14:20:38.030508 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: availability_zone_filter.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.030696 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Availability Zone Filter Extension" alias="availability_zone_filter" description="Add filter parameters to AvailabilityZone resource" updated="2018-06-22T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.030817 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension availability_zone_filter not supported by any of loaded plugins Jun 08 14:20:38.030886 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: data_plane_status.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.031081 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Port data plane status extension" alias="data-plane-status" description="Expose status of underlying data plane" updated="2017-01-24T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.031201 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension data-plane-status not supported by any of loaded plugins Jun 08 14:20:38.031283 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: default_subnetpools.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.031478 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Default Subnetpools" alias="default-subnetpools" description="Provides ability to mark and use a subnetpool as the default." updated="2016-02-18T18:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.031592 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: default-subnetpools Jun 08 14:20:38.031658 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: dhcpagentscheduler.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.031980 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="DHCP Agent Scheduler" alias="dhcp_agent_scheduler" description="Schedule networks among dhcp agents" updated="2013-02-07T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.032122 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: dhcp_agent_scheduler Jun 08 14:20:38.032189 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: dns.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.032402 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="DNS Integration" alias="dns-integration" description="Provides integration with DNS." updated="2015-08-15T18:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.032524 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: dns-integration Jun 08 14:20:38.032589 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: dns_domain_ports.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.032768 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="dns_domain for ports" alias="dns-domain-ports" description="Allows the DNS domain to be specified for a network port." updated="2017-04-24T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.032894 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: dns-domain-ports Jun 08 14:20:38.032959 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: dns_integration_domain_keywords.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.033153 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="DNS domain names with keywords allowed" alias="dns-integration-domain-keywords" description="Allows to use keywords like , , and as DNS domain name" updated="2020-06-15T18:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.033278 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: dns-integration-domain-keywords Jun 08 14:20:38.033384 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: dvr.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.033562 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Distributed Virtual Router" alias="dvr" description="Enables configuration of Distributed Virtual Routers." updated="2014-06-1T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.033677 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension dvr not supported by any of loaded plugins Jun 08 14:20:38.033742 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: ecmp_routes.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.033953 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Equal Cost Multipath Routing" alias="ecmp_routes" description="Allows traffic to reach thesame destination via multiple different links " updated="2020-11-26T18:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.034088 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension ecmp_routes not supported by any of loaded plugins Jun 08 14:20:38.034158 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: empty_string_filtering.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.034353 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Empty String Filtering Extension" alias="empty-string-filtering" description="Allow filtering by attributes with empty string value" updated="2018-05-01T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.034487 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: empty-string-filtering Jun 08 14:20:38.034557 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: evpn.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.034732 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="EVPN" alias="evpn" description="Router extension to support EVPN Type-5 route advertisement" updated="2026-04-28T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.034847 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension evpn not supported by any of loaded plugins Jun 08 14:20:38.034924 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: expose_l3_conntrack_helper.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.035110 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Expose CT target rules for conntrack helper" alias="expose-l3-conntrack-helper" description="Expose allow adding CT target rules for conntrack helper" updated="2019-04-04T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.035228 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension expose-l3-conntrack-helper not supported by any of loaded plugins Jun 08 14:20:38.035314 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: expose_port_forwarding_in_fip.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.035500 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Expose Floating IP port forwardings" alias="expose-port-forwarding-in-fip" description="Expose port forwarding fields in floating IP response" updated="2018-06-28T16:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.035618 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension expose-port-forwarding-in-fip not supported by any of loaded plugins Jun 08 14:20:38.035684 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: external_net.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.035914 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Neutron external network" alias="external-net" description="Adds external network attribute to network resource." updated="2013-01-14T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.035998 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: external-net Jun 08 14:20:38.036065 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: extra_dhcp_opt.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.036260 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Neutron Extra DHCP options" alias="extra_dhcp_opt" description="Extra options configuration for DHCP. For example PXE boot options to DHCP clients can be specified (e.g. tftp-server, server-ip-address, bootfile-name)" updated="2013-03-17T12:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.036385 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: extra_dhcp_opt Jun 08 14:20:38.036468 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: extraroute.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.036639 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Neutron Extra Route" alias="extraroute" description="Extra routes configuration for L3 router" updated="2013-02-01T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.036762 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: extraroute Jun 08 14:20:38.036844 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: extraroute_atomic.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.037047 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Atomically add/remove extra routes" alias="extraroute-atomic" description="Edit extra routes of a router on server side by atomically adding/removing extra routes" updated="2019-07-10T00:00:00+00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.037190 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: extraroute-atomic Jun 08 14:20:38.037294 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: filter_validation.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.037500 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Filter parameters validation" alias="filter-validation" description="Provides validation on filter parameters." updated="2018-07-04T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.037619 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: filter-validation Jun 08 14:20:38.037683 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: fip_pf_description.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.037900 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Floating IP Port Forwarding new attribute description" alias="floating-ip-port-forwarding-description" description="Add a description field to Port Forwarding rules" updated="2019-11-01T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.038020 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension floating-ip-port-forwarding-description not supported by any of loaded plugins Jun 08 14:20:38.038086 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: fip_pf_detail.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.038294 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Floating IP port forwarding detail" alias="floating-ip-port-forwarding-detail" description="Allow list floating ip return more port forwarding data, include'id' and 'internal_port_id'" updated="2022-05-06T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.038417 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: floating-ip-port-forwarding-detail Jun 08 14:20:38.038497 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: fip_pf_port_range.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.038671 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Floating IP Port Forwarding support port ranges" alias="floating-ip-port-forwarding-port-ranges" description="Add support to port ranges" updated="2020-07-01T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.038810 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension floating-ip-port-forwarding-port-ranges not supported by any of loaded plugins Jun 08 14:20:38.038889 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: fip_port_details.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.039065 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Floating IP Port Details Extension" alias="fip-port-details" description="Add port_details attribute to Floating IP resource" updated="2018-04-09T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.039208 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: fip-port-details Jun 08 14:20:38.039301 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: flavors.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.039527 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Neutron Service Flavors" alias="flavors" description="Flavor specification for Neutron advanced services." updated="2015-09-17T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.039640 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: flavors Jun 08 14:20:38.039708 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: floating_ip_port_forwarding.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.040006 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Floating IP Port Forwarding" alias="floating-ip-port-forwarding" description="Allow port forwarding from floating IP port to an internal port" updated="2018-01-19T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.040138 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension floating-ip-port-forwarding not supported by any of loaded plugins Jun 08 14:20:38.040221 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: floatingip_pools.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.040457 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Floating IP Pools Extension" alias="floatingip-pools" description="Provides a floating IP pools API." updated="2018-03-21T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.040592 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: floatingip-pools Jun 08 14:20:38.040658 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: ip_allocation.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.040868 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="IP Allocation" alias="ip_allocation" description="IP allocation extension." updated="2016-06-10T23:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.040997 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension ip_allocation not supported by any of loaded plugins Jun 08 14:20:38.041062 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: ip_substring_port_filtering.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.041263 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="IP address substring filtering" alias="ip-substring-filtering" description="Provides IP address substring filtering when listing ports" updated="2017-11-28T09:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.041455 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension ip-substring-filtering not supported by any of loaded plugins Jun 08 14:20:38.041455 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: l2_adjacency.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.041624 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="L2 Adjacency" alias="l2_adjacency" description="Display L2 Adjacency for Neutron Networks." updated="2016-04-12T16:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.041742 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension l2_adjacency not supported by any of loaded plugins Jun 08 14:20:38.041810 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: l3.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.042193 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Neutron L3 Router" alias="router" description="Router abstraction for basic L3 forwarding between L2 Neutron networks and access to external networks via a NAT gateway." updated="2012-07-20T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.042350 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: router Jun 08 14:20:38.042428 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: l3_agent_scheduler_ha_chassis_priority.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.042617 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="L3 Agent Scheduler HA Chassis Priority" alias="l3-agent-scheduler-ha-chassis-priority" description="Exposes the HA_Chassis priority in the L3 Agent Scheduler API responses and allows setting priority when scheduling a router to a chassis." updated="2026-03-31T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.042737 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: l3-agent-scheduler-ha-chassis-priority Jun 08 14:20:38.042825 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: l3_conntrack_helper.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.043374 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="L3 Conntrack helper" alias="l3-conntrack-helper" description="Allow adding CT target rules for conntrack helper" updated="2019-04-04T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.043509 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension l3-conntrack-helper not supported by any of loaded plugins Jun 08 14:20:38.043588 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: l3_enable_default_route_bfd.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.043781 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Enable Default Route BFD" alias="enable-default-route-bfd" description="Enables configurable BFD behavior for default routes" updated="2023-02-27T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.043956 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: enable-default-route-bfd Jun 08 14:20:38.044044 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: l3_enable_default_route_ecmp.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.044259 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Enable Default Route ECMP" alias="enable-default-route-ecmp" description="Enables configurable ECMP behavior for default routes" updated="2023-02-27T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.044390 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: enable-default-route-ecmp Jun 08 14:20:38.044458 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: l3_ext_gw_mode.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.044652 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Neutron L3 Configurable external gateway mode" alias="ext-gw-mode" description="Extension of the router abstraction for specifying whether SNAT should occur on the external gateway" updated="2013-03-28T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.044782 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: ext-gw-mode Jun 08 14:20:38.044857 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: l3_ext_ha_mode.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.045041 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="HA Router extension" alias="l3-ha" description="Adds HA capability to routers." updated="2014-04-26T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.045160 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: l3-ha Jun 08 14:20:38.045229 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: l3_ext_ndp_proxy.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.045429 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Router support to enable NDP proxy" alias="l3-ext-ndp-proxy" description="Router extension to enable ndp proxy" updated="2020-08-25T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.045542 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension l3-ext-ndp-proxy not supported by any of loaded plugins Jun 08 14:20:38.045620 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: l3_extra_gws.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.045917 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Neutron L3 External Gateway Multihoming" alias="external-gateway-multihoming" description="Allow multiple external gateway ports per router" updated="2023-01-18T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.046044 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: external-gateway-multihoming Jun 08 14:20:38.046109 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: l3_flavors.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.046320 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Router Flavor Extension" alias="l3-flavors" description="Flavor support for routers." updated="2016-05-17T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.046439 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: l3-flavors Jun 08 14:20:38.046510 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: l3_ndp_proxy.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.046747 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="name" alias="l3-ndp-proxy" description="description" updated="2021-06-29T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.046893 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension l3-ndp-proxy not supported by any of loaded plugins Jun 08 14:20:38.046968 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: l3_port_ip_change_not_allowed.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.047148 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Prevent L3 router ports IP address change extension" alias="l3-port-ip-change-not-allowed" description="Prevent change of IP address for some L3 router ports" updated="2018-10-09T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.047284 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension l3-port-ip-change-not-allowed not supported by any of loaded plugins Jun 08 14:20:38.047378 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: l3agentscheduler.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.047862 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="L3 Agent Scheduler" alias="l3_agent_scheduler" description="Schedule routers among l3 agents" updated="2013-02-07T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.048000 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: l3_agent_scheduler Jun 08 14:20:38.048066 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: local_ip.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.048337 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Local IP" alias="local_ip" description="Support Node Local Virtual IP" updated="2021-07-26T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.048458 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension local_ip not supported by any of loaded plugins Jun 08 14:20:38.048523 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: logging.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.048776 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Logging API Extension" alias="logging" description="Provides a logging API for resources such as security group and firewall." updated="2017-01-01T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.048923 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension logging not supported by any of loaded plugins Jun 08 14:20:38.048990 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: metering.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.049260 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Neutron Metering" alias="metering" description="Neutron Metering extension." updated="2013-06-12T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.049394 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension metering not supported by any of loaded plugins Jun 08 14:20:38.049460 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: metering_source_and_destination_fields.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.049668 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Neutron Metering with source and destination filtering" alias="metering_source_and_destination_fields" description="Neutron Metering extension that enables the use of source and destination IP prefixes to create metering label rules." updated="2020-08-21T08:12:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.049791 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension metering_source_and_destination_fields not supported by any of loaded plugins Jun 08 14:20:38.049857 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: multiprovidernet.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.050070 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Multi Provider Network" alias="multi-provider" description="Expose mapping of virtual networks to multiple physical networks" updated="2013-06-27T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.050187 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: multi-provider Jun 08 14:20:38.050264 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: netmtu.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.050451 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Network MTU" alias="net-mtu" description="Provides MTU attribute for a network resource." updated="2015-03-25T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.050559 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: net-mtu Jun 08 14:20:38.050624 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: netmtu_writable.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.050824 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Network MTU (writable)" alias="net-mtu-writable" description="Provides a writable MTU attribute for a network resource." updated="2017-07-12T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.050968 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: net-mtu-writable Jun 08 14:20:38.051040 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: network_availability_zone.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.051237 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Network Availability Zone" alias="network_availability_zone" description="Availability zone support for network." updated="2015-01-01T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.051375 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: network_availability_zone Jun 08 14:20:38.051440 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: network_ha.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.051613 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Network HA creation flag" alias="network_ha" description="Network high availability creation flag." updated="2023-04-27T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.051746 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension network_ha not supported by any of loaded plugins Jun 08 14:20:38.051812 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: network_ip_availability.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.052021 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Network IP Availability" alias="network-ip-availability" description="Provides IP availability data for each network and subnet." updated="2015-09-24T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.052137 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: network-ip-availability Jun 08 14:20:38.052201 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: network_ip_availability_details.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.052405 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Network IP Availability Details Extension" alias="network-ip-availability-details" description="Add ip_availability_details attribute to network-ip-availabilities resource." updated="2025-10-31T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.052522 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: network-ip-availability-details Jun 08 14:20:38.052588 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: network_segment_range.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.052863 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Neutron Network Segment Range" alias="network-segment-range" description="Provides support for the network segment range management" updated="2018-11-29T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.053012 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension network-segment-range not supported by any of loaded plugins Jun 08 14:20:38.053085 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: pagination.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.053287 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Pagination support" alias="pagination" description="Extension that indicates that pagination is enabled." updated="2016-06-12T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.053358 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: pagination Jun 08 14:20:38.053425 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: port_device_profile.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.053598 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Port device profile" alias="port-device-profile" description="Expose the port device profile (Cyborg)" updated="2020-12-17T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.053728 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: port-device-profile Jun 08 14:20:38.053795 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: port_hardware_offload_type.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.053989 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Port hardware offload" alias="port-hardware-offload-type" description="Expose port hardware offload extension" updated="2023-05-09T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.054109 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension port-hardware-offload-type not supported by any of loaded plugins Jun 08 14:20:38.054174 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: port_hint_ovs_tx_steering.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.054365 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Port hint: Open vSwitch Tx steering" alias="port-hint-ovs-tx-steering" description="Enable Port hint: Open vSwitch Tx steering" updated="2023-01-01T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.054479 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension port-hint-ovs-tx-steering not supported by any of loaded plugins Jun 08 14:20:38.054573 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: port_hints.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.054776 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Port hints" alias="port-hints" description="Backend specific port hints to allow backend specific tuning" updated="2023-01-01T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.054914 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension port-hints not supported by any of loaded plugins Jun 08 14:20:38.054982 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: port_mac_address_override.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.055170 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Neutron Port MAC address override" alias="port-mac-override" description="Allow overriding the MAC address of a direct-physical Port via the active binding profile" updated="2022-03-18T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.055300 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension port-mac-override not supported by any of loaded plugins Jun 08 14:20:38.055368 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: port_mac_address_regenerate.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.055560 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Neutron Port MAC address regenerate" alias="port-mac-address-regenerate" description="Network port MAC address regenerate" updated="2018-05-03T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.055677 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: port-mac-address-regenerate Jun 08 14:20:38.055756 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: port_numa_affinity_policy.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.055985 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Port NUMA affinity policy" alias="port-numa-affinity-policy" description="Expose the port NUMA affinity policy" updated="2020-07-08T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.056116 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: port-numa-affinity-policy Jun 08 14:20:38.056184 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: port_numa_affinity_policy_socket.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.056374 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Port NUMA affinity policy "socket"" alias="port-numa-affinity-policy-socket" description="Adds "socket" to the supported port NUMA affinity policies" updated="2024-02-09T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.056499 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: port-numa-affinity-policy-socket Jun 08 14:20:38.056591 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: port_resource_request.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.056781 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Port Resource Request" alias="port-resource-request" description="Expose resource request to Port" updated="2018-05-08T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.056912 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension port-resource-request not supported by any of loaded plugins Jun 08 14:20:38.056980 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: port_resource_request_groups.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.057178 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Port Resource Request Groups" alias="port-resource-request-groups" description="Support requesting multiple groups of resources and traits from the same RP subtree in resource_request" updated="2021-08-02T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.057318 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension port-resource-request-groups not supported by any of loaded plugins Jun 08 14:20:38.057408 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: port_trusted_vif.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.057588 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Port trusted vif" alias="port-trusted-vif" description="Expose port 'trusted' attribute in the API" updated="2024-07-10T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.057702 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension port-trusted-vif not supported by any of loaded plugins Jun 08 14:20:38.057783 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: portbindings.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.057996 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Port Binding" alias="binding" description="Expose port bindings of a virtual port to external application" updated="2014-02-03T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.058131 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: binding Jun 08 14:20:38.058199 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: portbindings_extended.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.058420 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Port Bindings Extended" alias="binding-extended" description="Expose port bindings of a virtual port to external application" updated="2017-07-17T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.058560 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: binding-extended Jun 08 14:20:38.058637 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: portsecurity.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.058825 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Port Security" alias="port-security" description="Provides port security" updated="2012-07-23T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.058954 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: port-security Jun 08 14:20:38.059018 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: project_id.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.059225 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="project_id field enabled" alias="project-id" description="Extension that indicates that project_id field is enabled." updated="2016-09-09T09:09:09-09:09" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.059318 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: project-id Jun 08 14:20:38.059384 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: providernet.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.059688 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Provider Network" alias="provider" description="Expose mapping of virtual networks to physical networks" updated="2012-09-07T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.059832 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: provider Jun 08 14:20:38.059915 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: pvlan.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.060106 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Private VLAN" alias="pvlan" description="The Private VLAN semantics extension." updated="2026-02-03T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.060232 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension pvlan not supported by any of loaded plugins Jun 08 14:20:38.060319 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: qinq.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.060537 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="QinQ networks" alias="qinq" description="Expose network 'qinq' attribute in the API. This allows to configure network to allow vlan in vlan configuration using 0x88a8 ethertype (QinQ)." updated="2024-12-02T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.060667 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: qinq Jun 08 14:20:38.060735 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: qos.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.061426 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Quality of Service" alias="qos" description="The Quality of Service extension." updated="2015-06-08T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.061553 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension qos not supported by any of loaded plugins Jun 08 14:20:38.061626 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: qos_bw_limit_direction.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.061848 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Direction for QoS bandwidth limit rule" alias="qos-bw-limit-direction" description="Allow to configure QoS bandwidth limit rule with specific direction: ingress or egress" updated="2017-04-10T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.061987 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension qos-bw-limit-direction not supported by any of loaded plugins Jun 08 14:20:38.062055 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: qos_bw_minimum_ingress.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.062258 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Ingress direction for QoS minimum bandwidth rule" alias="qos-bw-minimum-ingress" description="Allow to configure QoS minumum bandwidth rule with ingress direction." updated="2018-07-09T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.062388 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension qos-bw-minimum-ingress not supported by any of loaded plugins Jun 08 14:20:38.062457 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: qos_default.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.062662 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="QoS default policy" alias="qos-default" description="Expose the QoS default policy per project" updated="2017-041-06T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.062785 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension qos-default not supported by any of loaded plugins Jun 08 14:20:38.062867 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: qos_fip.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.063057 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Floating IP QoS" alias="qos-fip" description="The floating IP Quality of Service extension" updated="2017-07-20T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.063191 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension qos-fip not supported by any of loaded plugins Jun 08 14:20:38.063275 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: qos_fip_network_policy.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.063458 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="QoS floating IP network policy ID" alias="qos-fip-network-policy" description="Adds a the QoS network ID to the floating IP definition" updated="2021-11-15T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.063578 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension qos-fip-network-policy not supported by any of loaded plugins Jun 08 14:20:38.063680 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: qos_gateway_ip.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.063818 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Router gateway IP QoS" alias="qos-gateway-ip" description="The Router gateway IP Quality of Service extension" updated="2018-02-24T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.063958 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension qos-gateway-ip not supported by any of loaded plugins Jun 08 14:20:38.064023 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: qos_port_network_policy.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.064207 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="QoS port network policy ID" alias="qos-port-network-policy" description="Adds a the QoS network ID to the port definition" updated="2019-11-01T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.064343 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension qos-port-network-policy not supported by any of loaded plugins Jun 08 14:20:38.064418 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: qos_pps_minimum_rule.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.064622 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="QoS minimum packet rate rule" alias="qos-pps-minimum" description="Allow to configure QoS minimum packet rate rule." updated="2021-07-14T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.064741 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension qos-pps-minimum not supported by any of loaded plugins Jun 08 14:20:38.064809 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: qos_pps_minimum_rule_alias.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.065014 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="QoS minimum packet rate rule alias" alias="qos-pps-minimum-rule-alias" description="API to enable GET, PUT and DELETE operations on QoS minimum packet rate rule without specifying policy ID" updated="2021-10-22T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.065132 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension qos-pps-minimum-rule-alias not supported by any of loaded plugins Jun 08 14:20:38.065213 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: qos_pps_rule.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.065426 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="QoS Rule Type Packet per Second Extension" alias="qos-pps" description="Add QoS Rule Type Packet per Second" updated="2021-05-12T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.065541 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension qos-pps not supported by any of loaded plugins Jun 08 14:20:38.065602 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: qos_rule_type_details.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.065773 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Details of QoS rule types" alias="qos-rule-type-details" description="Expose details about QoS rule types supported by loaded backend drivers" updated="2017-06-22T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.065904 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension qos-rule-type-details not supported by any of loaded plugins Jun 08 14:20:38.065990 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: qos_rule_type_filter.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.066174 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Allow to filter the list of QoS rule types" alias="qos-rule-type-filter" description="Allows to filter the QoS rule type list adding two new flags. "all_rules" prints all implemented QoS rule types."all_supported" prints all supported QoS rule types by the loaded mechanism drivers" updated="2022-02-02T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.066323 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension qos-rule-type-filter not supported by any of loaded plugins Jun 08 14:20:38.066391 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: qos_rules_alias.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.066597 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Quality of Service rules alias API" alias="qos-rules-alias" description="API to enable GET, PUT and DELETE operations on QoS policy rules without specifying policy ID" updated="2018-10-07T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.066716 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension qos-rules-alias not supported by any of loaded plugins Jun 08 14:20:38.066781 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: quota_check_limit.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.066971 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Quota engine limit check" alias="quota-check-limit" description="Support for checking the resource usage before applying a new quota limit" updated="2021-09-08T16:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.067089 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: quota-check-limit Jun 08 14:20:38.067152 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: quota_check_limit_default.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.067353 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Quota engine limit check by default" alias="quota-check-limit-default" description="By default, the Neutron quota engine checks the resource usage before applying a new quota limit" updated="2024-08-21T16:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.067483 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: quota-check-limit-default Jun 08 14:20:38.067565 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: quotasv2.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.067974 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Quota management support" alias="quotas" description="Expose functions for quotas management per project" updated="2012-07-29T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.068047 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: quotas Jun 08 14:20:38.068110 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: quotasv2_detail.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.068402 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Quota details management support" alias="quota_details" description="Expose functions for quotas usage statistics per project" updated="2017-02-10T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.068474 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: quota_details Jun 08 14:20:38.068569 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: rbac.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.068986 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="RBAC Policies" alias="rbac-policies" description="Allows creation and modification of policies that control project access to resources." updated="2015-06-17T12:15:12-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.069125 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: rbac-policies Jun 08 14:20:38.069209 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: rbac_address_group.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.069427 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Add address_group type to RBAC" alias="rbac-address-group" description="Add address_group type to network RBAC" updated="2021-01-20T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.069550 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension rbac-address-group not supported by any of loaded plugins Jun 08 14:20:38.069615 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: rbac_address_scope.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.069792 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Add address_scope type to RBAC" alias="rbac-address-scope" description="Add address_scope type to RBAC" updated="2020-02-12T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.069917 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: rbac-address-scope Jun 08 14:20:38.069996 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: rbac_security_groups.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.070169 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Add security_group type to network RBAC" alias="rbac-security-groups" description="Add security_group type to network RBAC" updated="2019-02-14T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.070323 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: rbac-security-groups Jun 08 14:20:38.070395 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: rbac_subnetpool.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.070591 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Add subnetpool type to RBAC" alias="rbac-subnetpool" description="Add subnetpool type to RBAC" updated="2020-02-05T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.070710 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension rbac-subnetpool not supported by any of loaded plugins Jun 08 14:20:38.070778 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: revisionifmatch.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.070965 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="If-Match constraints based on revision_number" alias="revision-if-match" description="Extension indicating that If-Match based on revision_number is supported." updated="2016-12-11T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.071093 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: revision-if-match Jun 08 14:20:38.071161 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: revisions.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.071392 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Resource revision numbers" alias="standard-attr-revisions" description="This extension will display the revision number of neutron resources." updated="2016-04-11T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.071504 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: standard-attr-revisions Jun 08 14:20:38.071573 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: router_availability_zone.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.071768 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Router Availability Zone" alias="router_availability_zone" description="Availability zone support for router." updated="2015-01-01T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.071894 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: router_availability_zone Jun 08 14:20:38.071967 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: router_enable_snat.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.072150 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Exposes the router "enable_snat" read-only configuration flag" alias="router-enable-snat" description="Exposes the router "enable_snat" read-only configuration flag" updated="2025-09-18T18:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.072291 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: router-enable-snat Jun 08 14:20:38.072369 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: routerservicetype.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.072569 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Router Service Type" alias="router-service-type" description="Provides router service type" updated="2013-01-29T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.072689 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension router-service-type not supported by any of loaded plugins Jun 08 14:20:38.072756 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: security_groups_default_rules.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.073112 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Default rules for security groups" alias="security-groups-default-rules" description="Configure set of security group rules used as default rules for every new security group" updated="2022-12-19T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.073237 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: security-groups-default-rules Jun 08 14:20:38.073336 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: security_groups_normalized_cidr.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.073551 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Normalized CIDR field for security group rules" alias="security-groups-normalized-cidr" description="Add new field with normalized remote_ip_prefix cidr in SG rule" updated="2020-07-28T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.073634 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: security-groups-normalized-cidr Jun 08 14:20:38.073700 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: security_groups_port_filtering.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.073885 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Port filtering on security groups" alias="port-security-groups-filtering" description="Provides security groups filtering when listing ports" updated="2018-01-09T09:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.074029 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension port-security-groups-filtering not supported by any of loaded plugins Jun 08 14:20:38.074102 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: security_groups_remote_address_group.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.074308 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Remote address group id field for security group rules" alias="security-groups-remote-address-group" description="Add new field of remote address group id in SG rules" updated="2020-08-25T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.074434 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: security-groups-remote-address-group Jun 08 14:20:38.074512 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: security_groups_rules_belongs_to_default_sg.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.074730 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Security group rule belongs to the project's default security group" alias="security-groups-rules-belongs-to-default-sg" description="Flag to determine if the security group rule belongs to the project's default security group" updated="2023-05-23T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.074886 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: security-groups-rules-belongs-to-default-sg Jun 08 14:20:38.074956 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: security_groups_shared_filtering.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.075304 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Security group filtering on the shared field" alias="security-groups-shared-filtering" description="Support filtering security groups on the shared field" updated="2021-10-05T09:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.075445 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: security-groups-shared-filtering Jun 08 14:20:38.075517 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: securitygroup.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.076828 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="security-group" alias="security-group" description="The security groups extension." updated="2012-10-05T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.076983 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: security-group Jun 08 14:20:38.077057 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: segment.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.077339 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Segment" alias="segment" description="Segments extension." updated="2016-02-24T17:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.077487 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension segment not supported by any of loaded plugins Jun 08 14:20:38.077576 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: segments_peer_subnet_host_routes.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.077797 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Segments peer-subnet host routes" alias="segments-peer-subnet-host-routes" description="Add host routes to subnets on a routed network (segments)" updated="2018-06-12T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.077939 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension segments-peer-subnet-host-routes not supported by any of loaded plugins Jun 08 14:20:38.078008 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: servicetype.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.078220 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Neutron Service Type Management" alias="service-type" description="API for retrieving service providers for Neutron advanced services" updated="2013-01-20T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.078366 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: service-type Jun 08 14:20:38.078449 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: sorting.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.078662 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Sorting support" alias="sorting" description="Extension that indicates that sorting is enabled." updated="2016-06-12T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.078741 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: sorting Jun 08 14:20:38.078821 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: standard_attr_segment.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.079010 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Standard Attribute Segment Extension" alias="standard-attr-segment" description="Add standard attributes to Segment resource" updated="2018-04-09T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.079143 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension standard-attr-segment not supported by any of loaded plugins Jun 08 14:20:38.079229 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: standardattrdescription.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.079459 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="standard-attr-description" alias="standard-attr-description" description="Extension to add descriptions to standard attributes" updated="2016-02-10T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.079584 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: standard-attr-description Jun 08 14:20:38.079650 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: stateful_security_group.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.079841 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Stateful security group" alias="stateful-security-group" description="Indicates if the security group is stateful or not" updated="2019-11-26T09:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.079979 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: stateful-security-group Jun 08 14:20:38.080045 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: stdattrs_common.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.080226 np0000002315 neutron-ovn-maintenance-worker[88443]: WARNING neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Did not find expected name "Stdattrs_common" in /opt/stack/neutron/neutron/extensions/stdattrs_common.py Jun 08 14:20:38.080315 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: subnet_dns_publish_fixed_ip.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.080517 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Subnet DNS publish fixed IP" alias="subnet-dns-publish-fixed-ip" description="Support choosing to publish DNS records for IPs from a subnet" updated="2019-05-24T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.080659 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: subnet-dns-publish-fixed-ip Jun 08 14:20:38.080726 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: subnet_external_network.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.080932 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Subnet belongs to an external network" alias="subnet-external-network" description="Informs if the subnet belongs to an external network" updated="2024-02-05T18:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.081055 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: subnet-external-network Jun 08 14:20:38.081132 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: subnet_onboard.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.081345 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Subnet Onboard" alias="subnet_onboard" description="Provides support for onboarding subnets into subnet pools" updated="2018-12-18T09:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.081477 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: subnet_onboard Jun 08 14:20:38.081546 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: subnet_segmentid_writable.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.081720 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Subnet SegmentID (writable)" alias="subnet-segmentid-writable" description="Provides a writable segment_id attribute for a subnet resource." updated="2018-03-12T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.081869 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension subnet-segmentid-writable not supported by any of loaded plugins Jun 08 14:20:38.081950 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: subnet_service_types.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.082124 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Subnet service types" alias="subnet-service-types" description="Provides ability to set the subnet service_types field" updated="2016-03-15T18:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.082260 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: subnet-service-types Jun 08 14:20:38.082351 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: subnetallocation.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.082609 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Subnet Allocation" alias="subnet_allocation" description="Enables allocation of subnets from a subnet pool" updated="2015-03-30T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.082730 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: subnet_allocation Jun 08 14:20:38.082813 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: subnetpool_prefix_ops.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.083028 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Subnet Pool Prefix Operations" alias="subnetpool-prefix-ops" description="Provides support for adjusting the prefix list of subnet pools" updated="2019-02-08T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.083153 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: subnetpool-prefix-ops Jun 08 14:20:38.083229 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: tag_creation.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.083429 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Tag creation extension" alias="tag-creation" description="Allow to create multiple tags for a resource" updated="2024-23-07T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.083544 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: tag-creation Jun 08 14:20:38.083610 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: tag_ports_during_bulk_creation.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.083807 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Tag Ports During Bulk Creation" alias="tag-ports-during-bulk-creation" description="Allow to tag ports during bulk creation" updated="2019-12-29T19:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.083952 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension tag-ports-during-bulk-creation not supported by any of loaded plugins Jun 08 14:20:38.084018 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: tagging.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.084693 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Tag support for resources with standard attribute: port, subnet, subnetpool, network, security_group, router, floatingip, policy, trunk, network_segment_range" alias="standard-attr-tag" description="Enables to set tag on resources with standard attribute." updated="2017-01-01T00:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.084836 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: standard-attr-tag Jun 08 14:20:38.084921 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: timestamp.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.085123 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Resource timestamps" alias="standard-attr-timestamp" description="Adds created_at and updated_at fields to all Neutron resources that have Neutron standard attributes." updated="2016-09-12T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.085260 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: standard-attr-timestamp Jun 08 14:20:38.085345 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: trunk.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.085538 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Trunk Extension" alias="trunk" description="Provides support for trunk ports" updated="2016-01-01T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.085654 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension trunk not supported by any of loaded plugins Jun 08 14:20:38.085720 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: trunk_details.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.085937 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Trunk port details" alias="trunk-details" description="Expose trunk port details" updated="2016-01-01T10:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.086057 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension trunk-details not supported by any of loaded plugins Jun 08 14:20:38.086123 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: uplink_status_propagation.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.086323 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Uplink status propagation extension" alias="uplink-status-propagation" description="Provides ability to propagate port uplink status." updated="2018-05-31T18:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.086439 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension uplink-status-propagation not supported by any of loaded plugins Jun 08 14:20:38.086519 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: uplink_status_propagation_updatable.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.086708 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Uplink status propagation updatable extension" alias="uplink-status-propagation-updatable" description="Provides ability to update the port uplink status." updated="2024-09-03T18:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.086840 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Extension uplink-status-propagation-updatable not supported by any of loaded plugins Jun 08 14:20:38.086917 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loading extension file: vlantransparent.py {{(pid=88443) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 14:20:38.087258 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Ext name="Vlantransparent" alias="vlan-transparent" description="Provides Vlan Transparent Networks" updated="2015-03-23T09:00:00-00:00" {{(pid=88443) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 14:20:38.087394 np0000002315 neutron-ovn-maintenance-worker[88443]: INFO neutron.api.extensions [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Loaded extension: vlan-transparent Jun 08 14:20:38.091206 np0000002315 neutron-ovn-maintenance-worker[88443]: WARNING oslo_service.backend._threading.service [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Service MaintenanceWorker is not picklable with spawn; falling back to fork. Please make the service spawn-safe to avoid this fallback. Jun 08 14:20:38.091533 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Full set of CONF: {{(pid=88443) _load_service_manager_options /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/oslo_config_glue.py:59}} Jun 08 14:20:38.091613 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ******************************************************************************** {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3229}} Jun 08 14:20:38.091679 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Configuration options gathered from: {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3230}} Jun 08 14:20:38.091729 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] command line args: ['--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3231}} Jun 08 14:20:38.091868 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] config files: ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3232}} Jun 08 14:20:38.091945 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ================================================================================ {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3237}} Jun 08 14:20:38.092059 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] agent_down_time = 75 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.092143 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] allow_automatic_dhcp_failover = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.092222 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] allow_bulk = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.092425 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] allowed_conntrack_helpers = [{'amanda': 'tcp'}, {'ftp': 'tcp'}, {'h323': 'udp'}, {'h323': 'tcp'}, {'irc': 'tcp'}, {'netbios-ns': 'udp'}, {'pptp': 'tcp'}, {'sane': 'tcp'}, {'sip': 'udp'}, {'sip': 'tcp'}, {'snmp': 'udp'}, {'tftp': 'udp'}] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.092534 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] api_extensions_path = {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.092654 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] api_paste_config = api-paste.ini {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.092740 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] api_workers = 2 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.092823 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] auth_strategy = keystone {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.092913 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] base_mac = fa:16:3e:00:00:00 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.092995 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] client_socket_timeout = 900 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.093080 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] config_dir = [] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.093140 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] config_file = ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.093218 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] config_source = [] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.093335 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] control_exchange = neutron {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.093423 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] core_plugin = ml2 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.093504 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] debug = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.093600 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] default_availability_zones = [] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.093746 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] default_log_levels = ['amqp=WARN', 'boto=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'OFPHandler=INFO', 'OfctlService=INFO', 'os_ken.base.app_manager=INFO', 'os_ken.controller.controller=INFO'] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.093849 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] dhcp_agent_notification = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.093953 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] dhcp_agents_per_network = 1 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.094074 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] dhcp_lease_duration = 86400 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.094134 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] dhcp_load_type = networks {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.094212 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] dns_domain = openstacklocal {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.094313 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] dvr_base_mac = fa:16:3f:00:00:00 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.094403 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] enable_default_route_bfd = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.094482 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] enable_default_route_ecmp = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.094590 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] enable_dvr = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.094681 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] enable_new_agents = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.094759 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] enable_services_on_agents_with_admin_state_down = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.094835 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] enable_snat_by_default = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.094921 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] enable_traditional_dhcp = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.094994 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] external_dns_driver = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.095066 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] filter_validation = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.095150 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] global_physnet_mtu = 1430 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.095196 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] graceful_shutdown_timeout = 60 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.095367 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] host = np0000002315 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.095452 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] host_dvr_for_dhcp = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.095530 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] http_retries = 3 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.095626 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] instance_format = [instance: %(uuid)s] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.095713 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.095799 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ipam_driver = internal {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.095911 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ipv6_pd_enabled = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.095993 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] l3_ha = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.096075 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] l3_ha_net_cidr = 169.254.192.0/18 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.096155 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] l3_ha_network_physical_name = {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.096233 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] l3_ha_network_type = {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.096339 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] log_color = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.096433 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] log_config_append = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.096536 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.096639 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] log_dir = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.096726 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] log_file = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.096780 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] log_options = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.096859 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] log_rotate_interval = 1 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.096951 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] log_rotate_interval_type = days {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.097031 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] log_rotation_type = none {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.097090 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.097142 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.097223 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.097322 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.097395 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.097475 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] max_allowed_address_pair = 10 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.097550 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] max_dns_nameservers = 5 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.097629 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] max_header_line = 16384 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.097718 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] max_l3_agents_per_router = 3 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.097825 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] max_logfile_count = 30 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.097917 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] max_logfile_size_mb = 200 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.097995 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] max_routes = 30 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.098069 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] max_subnet_host_routes = 20 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.098147 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] my_ip = 10.4.3.119 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.098224 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] my_ipv6 = ::1 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.098317 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] network_auto_schedule = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.098407 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] network_link_prefix = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.098494 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.WeightScheduler {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.098600 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] notify_nova_on_port_data_changes = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.098696 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] notify_nova_on_port_status_changes = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.098779 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] pagination_max_limit = -1 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.098854 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] periodic_fuzzy_delay = 5 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.098939 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] periodic_interval = 40 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.099013 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] publish_errors = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.099089 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] rate_limit_burst = 0 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.099169 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] rate_limit_except_level = CRITICAL {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.099280 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] rate_limit_interval = 0 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.099362 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] router_distributed = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.099452 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] rpc_resources_processing_step = 20 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.099528 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] rpc_response_max_timeout = 600 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.099605 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] rpc_response_timeout = 60 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.099702 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] rpc_state_report_workers = 0 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.099782 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] rpc_workers = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.099859 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] send_events_interval = 2 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.099958 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] service_plugins = ['ovn-router'] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.100039 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] setproctitle = on {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.100111 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] shell_completion = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.100189 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] state_path = /opt/stack/data/neutron {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.100283 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] syslog_log_facility = LOG_USER {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.100361 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] tcp_keepidle = 600 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.100454 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] transport_url = **** {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.100559 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] use_journal = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.100643 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] use_json = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.100733 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] use_stderr = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.100816 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] use_syslog = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.100902 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] vlan_qinq = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.100991 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] vlan_transparent = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.101082 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] wsgi_default_pool_size = 100 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.101160 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] wsgi_keep_alive = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.101241 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] wsgi_log_format = %(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.101336 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] wsgi_server_debug = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.101443 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_concurrency.disable_process_locking = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.101536 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_concurrency.lock_path = /opt/stack/data/neutron/lock {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.101626 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] profiler.connection_string = messaging:// {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.101718 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] profiler.enabled = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.101809 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] profiler.es_doc_type = notification {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.101896 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] profiler.es_scroll_size = 10000 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.101979 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] profiler.es_scroll_time = 2m {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.102058 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] profiler.filter_error_trace = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.102140 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] profiler.hmac_keys = **** {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.102219 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] profiler.sentinel_service_name = mymaster {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.102328 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] profiler.socket_timeout = 0.1 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.102408 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] profiler.trace_requests = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.102508 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] profiler.trace_sqlalchemy = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.102620 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] profiler_jaeger.process_tags = {} {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.102700 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] profiler_jaeger.service_name_prefix = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.102796 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] profiler_otlp.service_name_prefix = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.102887 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.102968 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.103048 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_metrics.metrics_process_name = {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.103128 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.103202 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.103306 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_policy.enforce_new_defaults = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.103489 np0000002315 neutron-ovn-maintenance-worker[88443]: WARNING oslo_config.cfg [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Deprecated: Option "enforce_scope" from group "oslo_policy" is deprecated for removal (This configuration was added temporarily to facilitate a smooth transition to the new RBAC. OpenStack will always enforce scope checks. This configuration option is deprecated and will be removed in the 2025.2 cycle.). Its value may be silently ignored in the future. Jun 08 14:20:38.103581 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_policy.enforce_scope = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.103667 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_policy.policy_default_rule = default {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.103772 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.103861 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_policy.policy_file = /etc/neutron/policy.json {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.103959 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.104037 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.104127 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.104224 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.104286 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.104368 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_policy.remote_timeout = 60.0 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.104457 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.104581 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] service_providers.service_provider = [] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.104673 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.auth_section = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.104780 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.auth_type = password {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.104879 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.auth_url = https://10.4.3.119/identity {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.104962 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.cafile = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.105039 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.certfile = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.105118 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.collect_timing = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.105194 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.default_domain_id = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.105283 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.default_domain_name = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.105363 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.domain_id = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.105436 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.domain_name = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.105531 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.endpoint_type = public {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.105610 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.insecure = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.105684 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.keyfile = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.105772 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.password = **** {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.105857 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.project_domain_id = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.105948 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.project_domain_name = Default {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.106028 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.project_id = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.106110 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.project_name = service {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.106189 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.region_name = RegionOne {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.106280 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.split_loggers = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.106360 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.system_scope = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.106435 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.timeout = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.106523 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.trust_id = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.106602 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.user_domain_id = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.106681 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.user_domain_name = Default {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.106754 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.user_id = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.106852 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] nova.username = nova {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.106946 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] placement.auth_section = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.107027 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] placement.auth_type = password {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.107102 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] placement.cafile = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.107176 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] placement.certfile = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.107264 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] placement.collect_timing = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.107355 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] placement.endpoint_type = public {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.107433 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] placement.insecure = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.107512 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] placement.keyfile = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.107596 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] placement.region_name = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.107672 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] placement.split_loggers = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.107745 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] placement.timeout = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.107850 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.auth_section = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.107963 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.auth_type = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.108043 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.cafile = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.108119 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.certfile = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.108197 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.collect_timing = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.108287 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.connect_retries = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.108363 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.connect_retry_delay = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.108439 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.enable_notifications = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.108518 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.endpoint_override = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.108605 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.insecure = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.108680 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.interface = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.108752 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.keyfile = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.108839 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.max_version = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.108929 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.min_version = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.109003 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.region_name = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.109077 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.retriable_status_codes = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.109150 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.service_name = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.109239 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.service_type = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.109336 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.split_loggers = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.109412 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.status_code_retries = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.109484 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.status_code_retry_delay = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.109572 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.timeout = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.109649 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.valid_interfaces = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.109722 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ironic.version = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.109818 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] cli_script.dry_run = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.109932 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.110013 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.110090 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.110168 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.110244 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.110341 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.110420 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.heartbeat_rate = 3 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.110498 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.110593 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.hostname = np0000002315 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.110674 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.110756 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.110835 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.110924 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.111001 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.kombu_reconnect_splay = 0.0 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.111082 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.processname = neutron-ovn-maintenance-worker {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.111159 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.111235 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.111332 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.111410 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.111483 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.111559 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_bytes = 0 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.111650 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_length = 0 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.111726 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.111818 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.111924 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.112003 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.rabbit_stream_fanout = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.112079 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.112153 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.rabbit_transient_quorum_queue = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.112226 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.112319 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.ssl = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.112400 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.112478 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.112553 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.112641 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.ssl_enforce_hostname_verification = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.112723 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.112801 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.ssl_version = {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.112882 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_rabbit.use_queue_manager = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.112964 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_notifications.driver = [] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.113040 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_notifications.retry = -1 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.113128 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.113201 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_messaging_notifications.transport_url = **** {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.113297 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_reports.file_event_handler = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.113375 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_reports.file_event_handler_interval = 1 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.113449 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_reports.log_dir = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.113527 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.113620 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] QUOTAS.default_quota = -1 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.113708 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] QUOTAS.quota_driver = neutron.db.quota.driver_nolock.DbQuotaNoLockDriver {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.113787 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] QUOTAS.quota_floatingip = 50 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.113896 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] QUOTAS.quota_network = 100 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.113977 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] QUOTAS.quota_port = 500 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.114054 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] QUOTAS.quota_rbac_policy = 10 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.114129 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] QUOTAS.quota_router = 10 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.114204 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] QUOTAS.quota_security_group = 10 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.114314 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] QUOTAS.quota_security_group_rule = 100 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.114373 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] QUOTAS.quota_subnet = 100 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.114448 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] QUOTAS.track_quota_usage = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.114531 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep.capabilities = [21, 12, 1, 2, 19] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.114612 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep.group = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.114699 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep.helper_command = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.114777 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep.log_daemon_traceback = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.114858 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep.logger_name = oslo_privsep.daemon {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.114945 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep.thread_pool_size = 12 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.115019 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep.user = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.115160 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_dhcp_release.capabilities = [21, 12] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.115231 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_dhcp_release.group = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.115331 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_dhcp_release.helper_command = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.115413 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_dhcp_release.log_daemon_traceback = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.115496 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_dhcp_release.logger_name = oslo_privsep.daemon {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.115569 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_dhcp_release.thread_pool_size = 12 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.115657 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_dhcp_release.user = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.115744 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_ovs_vsctl.capabilities = [21, 12] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.115846 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_ovs_vsctl.group = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.115941 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_ovs_vsctl.helper_command = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.116021 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_ovs_vsctl.log_daemon_traceback = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.116100 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_ovs_vsctl.logger_name = oslo_privsep.daemon {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.116202 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_ovs_vsctl.thread_pool_size = 12 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.116242 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_ovs_vsctl.user = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.116337 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_namespace.capabilities = [21] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.116413 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_namespace.group = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.116485 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_namespace.helper_command = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.116560 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_namespace.log_daemon_traceback = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.116642 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_namespace.logger_name = oslo_privsep.daemon {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.116728 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_namespace.thread_pool_size = 12 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.116798 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_namespace.user = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.116884 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_conntrack.capabilities = [12] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.116962 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_conntrack.group = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.117035 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_conntrack.helper_command = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.117108 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_conntrack.log_daemon_traceback = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.117184 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_conntrack.logger_name = oslo_privsep.daemon {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.117259 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_conntrack.thread_pool_size = 12 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.117349 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_conntrack.user = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.117432 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_link.capabilities = [12, 21] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.117504 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_link.group = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.117575 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_link.helper_command = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.117655 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_link.log_daemon_traceback = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.117743 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_link.logger_name = oslo_privsep.daemon {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.117845 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_link.thread_pool_size = 12 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.117951 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] privsep_link.user = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.118053 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] AGENT.root_helper = sudo /opt/stack/data/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.118139 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] AGENT.root_helper_daemon = sudo /opt/stack/data/venv/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.118216 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] AGENT.use_helper_for_ns_read = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.118314 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] SECURITYGROUP.enable_ipset = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.118399 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] SECURITYGROUP.enable_security_group = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.118472 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] SECURITYGROUP.firewall_driver = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.118551 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] SECURITYGROUP.permitted_ethertypes = [] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.118630 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] experimental.ipv6_pd_enabled = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.118737 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ml2.extension_drivers = ['port_security', 'qos'] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.118815 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ml2.external_network_type = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.118921 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ml2.mechanism_drivers = ['ovn'] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.119008 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ml2.overlay_ip_version = 4 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.119085 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ml2.path_mtu = 0 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.119165 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ml2.physical_network_mtus = {} {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.119263 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ml2.project_network_types = ['geneve'] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.119351 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ml2.tunnelled_network_rp_name = rp_tunnelled {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.119438 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ml2.type_drivers = ['local', 'flat', 'vlan', 'geneve'] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.119526 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ml2_type_flat.flat_networks = ['public'] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.119610 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ml2_type_geneve.max_header_size = 38 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.119705 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ml2_type_geneve.vni_ranges = ['1:65536'] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.119806 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ml2_type_vlan.network_vlan_ranges = ['public'] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.119917 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] network_log.burst_limit = 25 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.119995 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] network_log.local_output_log_base = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.120072 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] network_log.rate_limit = 100 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.120154 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.asyncio_connection = **** {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.120228 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.asyncio_slave_connection = **** {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.120326 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.backend = sqlalchemy {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.120406 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.connection = **** {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.120481 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.connection_debug = 0 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.120558 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.connection_parameters = {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.120635 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.connection_recycle_time = 3600 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.120722 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.connection_trace = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.120805 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.db_inc_retry_interval = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.120890 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.db_max_retries = 20 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.120965 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.db_max_retry_interval = 10 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.121037 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.db_retry_interval = 1 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.121110 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.max_overflow = 50 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.121183 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.max_pool_size = 5 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.121262 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.max_retries = 10 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.121353 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.mysql_sql_mode = TRADITIONAL {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.121428 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.mysql_wsrep_sync_wait = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.121500 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.pool_timeout = None {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.121574 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.retry_interval = 10 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.121645 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.slave_connection = **** {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.121728 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.sqlite_synchronous = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.121832 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.synchronous_reader = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.121930 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] database.use_db_reconnect = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.122028 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.bfd_min_rx = 1000 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.122108 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.bfd_min_tx = 100 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.122183 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.bfd_mult = 3 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.122260 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.broadcast_arps_to_all_routers = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.122352 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.dhcp_default_lease_time = 43200 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.122429 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.disable_ovn_dhcp_for_baremetal_ports = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.122504 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.dns_records_ovn_owned = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.122584 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.dns_servers = [] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.122660 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.enable_distributed_floating_ip = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.122750 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.fdb_age_threshold = 300 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.122837 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.ha_failover_strategy = normal {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.122930 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.live_migration_activation_strategy = rarp {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.123007 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.localnet_learn_fdb = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.123083 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.mac_binding_age_threshold = 300 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.123164 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.neutron_sync_mode = log {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.123241 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.ovn_dhcp4_global_options = {} {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.123336 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.ovn_dhcp6_global_options = {} {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.123419 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.ovn_l3_scheduler = leastloaded {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.123498 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.ovn_metadata_enabled = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.123576 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.ovn_nb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.123654 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.ovn_nb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.123746 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.ovn_nb_connection = ['ssl:10.4.3.119:6641'] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.123859 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.ovn_nb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.123956 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.ovn_router_indirect_snat = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.124038 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.ovn_sb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.124115 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.ovn_sb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.124199 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.ovn_sb_connection = ['ssl:10.4.3.119:6642'] {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.124293 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.ovn_sb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.124415 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.ovs_create_tap = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.124461 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.ovsdb_connection_timeout = 180 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.124527 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.ovsdb_log_level = INFO {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.124604 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.ovsdb_probe_interval = 60000 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.124676 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.ovsdb_retry_max_interval = 180 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.124757 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.stateless_nat_enabled = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.124849 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn.vhost_sock_dir = /var/run/openvswitch {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.124943 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] OVS.bridge_mac_table_size = 50000 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.125019 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] OVS.igmp_flood = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.125094 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] OVS.igmp_flood_reports = True {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.125169 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] OVS.igmp_flood_unregistered = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.125258 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] OVS.igmp_snooping_enable = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.125343 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] OVS.ovsdb_timeout = 10 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.125423 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn_nb_global.fdb_removal_limit = 0 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.125501 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn_nb_global.ignore_lsp_down = False {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.125577 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ovn_nb_global.mac_binding_removal_limit = 0 {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.125632 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG cotyledon.oslo_config_glue [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] ******************************************************************************** {{(pid=88443) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3263}} Jun 08 14:20:38.125710 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG oslo_service.backend._threading.service [None req-348b2523-c43c-47c5-b1d2-35aa39c6460d None None] Selected the multiprocessing context: and updated it in Cotyledon ServiceManager: {{(pid=88443) launch_service /opt/stack/data/venv/lib/python3.12/site-packages/oslo_service/backend/_threading/service.py:405}} Jun 08 14:20:38.144890 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] Full set of CONF: {{(pid=89274) _load_service_options /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/oslo_config_glue.py:65}} Jun 08 14:20:38.145310 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ******************************************************************************** {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3229}} Jun 08 14:20:38.145310 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] Configuration options gathered from: {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3230}} Jun 08 14:20:38.145310 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] command line args: ['--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3231}} Jun 08 14:20:38.145467 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] config files: ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3232}} Jun 08 14:20:38.145467 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ================================================================================ {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3237}} Jun 08 14:20:38.145561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] agent_down_time = 75 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.145656 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] allow_automatic_dhcp_failover = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.145750 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] allow_bulk = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.145970 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] allowed_conntrack_helpers = [{'amanda': 'tcp'}, {'ftp': 'tcp'}, {'h323': 'udp'}, {'h323': 'tcp'}, {'irc': 'tcp'}, {'netbios-ns': 'udp'}, {'pptp': 'tcp'}, {'sane': 'tcp'}, {'sip': 'udp'}, {'sip': 'tcp'}, {'snmp': 'udp'}, {'tftp': 'udp'}] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.146059 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] api_extensions_path = {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.146141 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] api_paste_config = api-paste.ini {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.146242 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] api_workers = 2 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.146332 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] auth_strategy = keystone {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.146414 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] base_mac = fa:16:3e:00:00:00 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.146505 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] client_socket_timeout = 900 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.146611 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] config_dir = [] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.146671 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] config_file = ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.146761 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] config_source = [] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.146847 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] control_exchange = neutron {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.146943 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] core_plugin = ml2 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.147022 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] debug = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.147108 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] default_availability_zones = [] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.147296 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] default_log_levels = ['amqp=WARN', 'boto=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'OFPHandler=INFO', 'OfctlService=INFO', 'os_ken.base.app_manager=INFO', 'os_ken.controller.controller=INFO'] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.147395 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] dhcp_agent_notification = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.147560 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] dhcp_agents_per_network = 1 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.147560 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] dhcp_lease_duration = 86400 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.147658 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] dhcp_load_type = networks {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.147699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] dns_domain = openstacklocal {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.147778 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] dvr_base_mac = fa:16:3f:00:00:00 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.147858 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] enable_default_route_bfd = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.147935 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] enable_default_route_ecmp = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.148009 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] enable_dvr = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.148081 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] enable_new_agents = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.148148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] enable_services_on_agents_with_admin_state_down = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.148225 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] enable_snat_by_default = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.148312 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] enable_traditional_dhcp = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.148383 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] external_dns_driver = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.148423 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] fatal_deprecations = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.148480 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] filter_validation = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.148589 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] global_physnet_mtu = 1430 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.148638 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] graceful_shutdown_timeout = 60 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.148901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] host = np0000002315 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.148979 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] host_dvr_for_dhcp = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.149049 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] http_retries = 3 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.149115 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] instance_format = [instance: %(uuid)s] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.149200 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.149272 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ipam_driver = internal {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.149336 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ipv6_pd_enabled = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.149413 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] l3_ha = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.149489 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_net_cidr = 169.254.192.0/18 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.149560 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_network_physical_name = {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.149638 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_network_type = {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.149716 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] log_color = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.149801 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] log_config_append = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.149882 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.149954 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] log_dir = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.150014 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] log_file = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.150061 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] log_options = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.150100 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] log_rotate_interval = 1 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.150174 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] log_rotate_interval_type = days {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.150245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] log_rotation_type = none {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.150338 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.150380 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.150418 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.150488 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.150539 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.150579 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] max_allowed_address_pair = 10 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.150644 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] max_dns_nameservers = 5 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.150715 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] max_header_line = 16384 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.150780 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] max_l3_agents_per_router = 3 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.150839 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] max_logfile_count = 30 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.150905 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] max_logfile_size_mb = 200 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.150972 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] max_routes = 30 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.151041 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] max_subnet_host_routes = 20 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.151112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] my_ip = 10.4.3.119 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.151188 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] my_ipv6 = ::1 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.151283 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] network_auto_schedule = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.151356 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] network_link_prefix = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.151436 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.WeightScheduler {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.151507 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] notify_nova_on_port_data_changes = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.151583 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] notify_nova_on_port_status_changes = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.151659 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] pagination_max_limit = -1 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.151760 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] periodic_fuzzy_delay = 5 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.151816 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] periodic_interval = 40 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.151910 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] publish_errors = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.151979 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_burst = 0 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.152054 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_except_level = CRITICAL {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.152113 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_interval = 0 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.152172 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] router_distributed = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.152241 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] rpc_resources_processing_step = 20 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.152351 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] rpc_response_max_timeout = 600 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.152436 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] rpc_response_timeout = 60 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.152502 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] rpc_state_report_workers = 0 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.152558 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] rpc_workers = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.152622 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] send_events_interval = 2 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.152705 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] service_plugins = ['ovn-router'] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.152769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] setproctitle = on {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.152841 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] shell_completion = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.152922 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] state_path = /opt/stack/data/neutron {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.152986 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] syslog_log_facility = LOG_USER {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.153058 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] tcp_keepidle = 600 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.153134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] transport_url = **** {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.153195 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] use_journal = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.153266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] use_json = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.153332 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] use_stderr = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.153405 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] use_syslog = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.153457 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] vlan_qinq = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.153516 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] vlan_transparent = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.153585 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] wsgi_default_pool_size = 100 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.153651 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] wsgi_keep_alive = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.153796 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] wsgi_log_format = %(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.153881 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] wsgi_server_debug = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 14:20:38.154000 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_concurrency.disable_process_locking = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.154097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_concurrency.lock_path = /opt/stack/data/neutron/lock {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.154198 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] profiler.connection_string = messaging:// {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.154287 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] profiler.enabled = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.154358 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_doc_type = notification {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.154431 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_scroll_size = 10000 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.154495 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_scroll_time = 2m {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.154557 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] profiler.filter_error_trace = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.154629 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] profiler.hmac_keys = **** {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.154696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] profiler.sentinel_service_name = mymaster {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.154782 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] profiler.socket_timeout = 0.1 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.154843 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] profiler.trace_requests = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.154922 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] profiler.trace_sqlalchemy = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.155033 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] profiler_jaeger.process_tags = {} {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.155097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] profiler_jaeger.service_name_prefix = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.155170 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] profiler_otlp.service_name_prefix = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.155263 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.155352 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.155425 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_process_name = {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.155509 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.155577 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.155690 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.enforce_new_defaults = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.155730 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.enforce_scope = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.155769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_default_rule = default {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.155831 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.155870 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_file = /etc/neutron/policy.json {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.155938 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.155990 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156068 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156113 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_timeout = 60.0 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156190 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156280 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] service_providers.service_provider = [] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156349 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_section = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_type = password {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156426 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_url = https://10.4.3.119/identity {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156484 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.cafile = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.certfile = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156576 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.collect_timing = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156615 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.default_domain_id = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.default_domain_name = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.domain_id = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.domain_name = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.endpoint_type = public {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156806 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.insecure = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156843 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.keyfile = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156889 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.password = **** {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156938 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.project_domain_id = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.156976 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.project_domain_name = Default {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157014 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.project_id = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157014 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.project_name = service {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157081 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.region_name = RegionOne {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.split_loggers = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157155 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.system_scope = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157192 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.timeout = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157230 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.trust_id = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157292 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.user_domain_id = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.user_domain_name = Default {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157369 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.user_id = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157406 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] nova.username = nova {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157451 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] placement.auth_section = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157494 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] placement.auth_type = password {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157532 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] placement.cafile = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157571 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] placement.certfile = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] placement.collect_timing = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157665 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] placement.endpoint_type = public {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157665 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] placement.insecure = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] placement.keyfile = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] placement.region_name = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157806 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] placement.split_loggers = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157806 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] placement.timeout = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157887 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.auth_section = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157927 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.auth_type = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.157965 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.cafile = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158002 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.certfile = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158049 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.collect_timing = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158092 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.connect_retries = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158177 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.connect_retry_delay = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158177 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.enable_notifications = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.endpoint_override = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.insecure = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158346 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.interface = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158346 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.keyfile = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158411 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.max_version = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158411 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.min_version = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158491 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.region_name = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158491 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.retriable_status_codes = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158555 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.service_name = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158555 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.service_type = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.split_loggers = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.status_code_retries = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158685 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.status_code_retry_delay = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158685 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.timeout = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158953 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.valid_interfaces = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.158997 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ironic.version = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159039 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] cli_script.dry_run = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159102 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159150 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159190 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159229 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159289 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159333 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159380 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.heartbeat_rate = 3 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159380 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159447 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.hostname = np0000002315 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159447 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_compression = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159511 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159549 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159549 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159616 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_reconnect_splay = 0.0 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159654 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.processname = neutron-ovn-maintenance-worker {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159841 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159841 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_max_memory_bytes = 0 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159920 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_max_memory_length = 0 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159920 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.159987 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160038 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160076 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_stream_fanout = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160114 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160159 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_transient_quorum_queue = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160197 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160235 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160292 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_ca_file = {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160330 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_cert_file = {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160384 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160429 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_enforce_hostname_verification = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160467 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_key_file = {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160467 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_version = {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160540 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.use_queue_manager = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160579 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.driver = [] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160616 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.retry = -1 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160659 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.topics = ['notifications'] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.transport_url = **** {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160739 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.file_event_handler = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160779 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.file_event_handler_interval = 1 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160779 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.log_dir = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160844 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160917 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.default_quota = -1 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.160958 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_driver = neutron.db.quota.driver_nolock.DbQuotaNoLockDriver {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_floatingip = 50 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161048 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_network = 100 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161098 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_port = 500 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161145 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_rbac_policy = 10 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161183 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_router = 10 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161222 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_security_group = 10 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161276 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_security_group_rule = 100 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161329 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_subnet = 100 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161368 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.track_quota_usage = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161430 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep.capabilities = [21, 12, 1, 2, 19] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep.group = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161507 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep.helper_command = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161545 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep.log_daemon_traceback = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161594 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep.logger_name = oslo_privsep.daemon {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161639 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep.thread_pool_size = 12 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161685 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep.user = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161723 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.capabilities = [21, 12] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161761 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.group = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161761 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.helper_command = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161828 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.log_daemon_traceback = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161865 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.logger_name = oslo_privsep.daemon {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161865 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.thread_pool_size = 12 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.user = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.161959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.capabilities = [21, 12] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162026 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.group = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162064 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.helper_command = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162101 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.log_daemon_traceback = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162146 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.logger_name = oslo_privsep.daemon {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162184 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.thread_pool_size = 12 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162184 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.user = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.capabilities = [21] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162313 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.group = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162351 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.helper_command = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.log_daemon_traceback = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162428 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.logger_name = oslo_privsep.daemon {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162464 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.thread_pool_size = 12 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162501 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.user = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162539 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.capabilities = [12] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162576 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.group = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162576 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.helper_command = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162651 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.log_daemon_traceback = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162651 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.logger_name = oslo_privsep.daemon {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162716 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.thread_pool_size = 12 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162716 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.user = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.capabilities = [12, 21] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162947 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.group = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162987 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.helper_command = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.162987 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.log_daemon_traceback = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163051 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.logger_name = oslo_privsep.daemon {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163051 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.thread_pool_size = 12 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163127 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.user = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163164 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] AGENT.root_helper = sudo /opt/stack/data/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163203 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] AGENT.root_helper_daemon = sudo /opt/stack/data/venv/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163203 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] AGENT.use_helper_for_ns_read = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163293 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.enable_ipset = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.enable_security_group = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163369 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.firewall_driver = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.permitted_ethertypes = [] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163461 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] experimental.ipv6_pd_enabled = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ml2.extension_drivers = ['port_security', 'qos'] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ml2.external_network_type = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ml2.mechanism_drivers = ['ovn'] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ml2.overlay_ip_version = 4 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163674 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ml2.path_mtu = 0 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163710 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ml2.physical_network_mtus = {} {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163749 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ml2.project_network_types = ['geneve'] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163786 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ml2.tunnelled_network_rp_name = rp_tunnelled {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163902 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ml2.type_drivers = ['local', 'flat', 'vlan', 'geneve'] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.163990 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_flat.flat_networks = ['public'] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164050 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_geneve.max_header_size = 38 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164088 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_geneve.vni_ranges = ['1:65536'] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164146 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_vlan.network_vlan_ranges = ['public'] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164201 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] network_log.burst_limit = 25 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164239 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] network_log.local_output_log_base = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164290 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] network_log.rate_limit = 100 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164357 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.asyncio_connection = **** {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164406 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.asyncio_slave_connection = **** {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164444 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.backend = sqlalchemy {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164483 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.connection = **** {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164528 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.connection_debug = 0 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164528 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.connection_parameters = {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164595 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.connection_recycle_time = 3600 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164631 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.connection_trace = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164669 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.db_inc_retry_interval = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164706 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.db_max_retries = 20 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164706 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.db_max_retry_interval = 10 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164785 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.db_retry_interval = 1 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164785 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.max_overflow = 50 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164851 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.max_pool_size = 5 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164851 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.max_retries = 10 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.mysql_sql_mode = TRADITIONAL {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.mysql_wsrep_sync_wait = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164995 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.pool_timeout = None {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.164995 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.retry_interval = 10 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165067 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.slave_connection = **** {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165067 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.sqlite_synchronous = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165147 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.synchronous_reader = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165147 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] database.use_db_reconnect = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165213 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_min_rx = 1000 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165273 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_min_tx = 100 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165315 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_mult = 3 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165351 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.broadcast_arps_to_all_routers = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.dhcp_default_lease_time = 43200 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.disable_ovn_dhcp_for_baremetal_ports = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165454 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.dns_records_ovn_owned = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165454 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.dns_servers = [] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165519 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.enable_distributed_floating_ip = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165557 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.fdb_age_threshold = 300 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165557 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.ha_failover_strategy = normal {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165636 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.live_migration_activation_strategy = rarp {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165636 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.localnet_learn_fdb = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165701 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.mac_binding_age_threshold = 300 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165701 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.neutron_sync_mode = log {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165766 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_dhcp4_global_options = {} {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165766 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_dhcp6_global_options = {} {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165839 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_l3_scheduler = leastloaded {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165839 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_metadata_enabled = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165915 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165960 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.165999 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_connection = ['ssl:10.4.3.119:6641'] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166037 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166074 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_router_indirect_snat = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166113 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166150 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166187 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_connection = ['ssl:10.4.3.119:6642'] {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166225 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166290 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovs_create_tap = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_connection_timeout = 180 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_log_level = INFO {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166403 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_probe_interval = 60000 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166403 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_retry_max_interval = 180 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.stateless_nat_enabled = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn.vhost_sock_dir = /var/run/openvswitch {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166531 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] OVS.bridge_mac_table_size = 50000 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166568 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166614 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood_reports = True {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166614 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood_unregistered = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166679 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_snooping_enable = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166679 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] OVS.ovsdb_timeout = 10 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166744 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.fdb_removal_limit = 0 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166788 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.ignore_lsp_down = False {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166827 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.mac_binding_removal_limit = 0 {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 14:20:38.166827 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon.oslo_config_glue [-] ******************************************************************************** {{(pid=89274) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3263}} Jun 08 14:20:38.167001 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG cotyledon._service_worker [-] Run service ServiceWrapper(0) [89274] {{(pid=89274) wait_forever /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/_service_worker.py:218}} Jun 08 14:20:38.167841 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-306338', 'neutron.services.pvlan.drivers.ovn.driver._initialize_pvlan_pg_drop-8755138364720', 'neutron.services.ovn_l3.plugin.OVNL3RouterPlugin._post_fork_initialize-537721'] for process (None), after_init {{(pid=89274) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Jun 08 14:20:38.169089 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Getting OvsdbNbOvnIdl for MaintenanceWorker with retry Jun 08 14:20:38.218170 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Logical_Switch.name {{(pid=89274) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 08 14:20:38.218295 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Logical_Router.name {{(pid=89274) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 08 14:20:38.218339 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Load_Balancer.name {{(pid=89274) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 08 14:20:38.218388 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Copp.name {{(pid=89274) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 14:20:38.218444 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Meter.name {{(pid=89274) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 14:20:38.218487 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Load_Balancer_Group.name {{(pid=89274) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 14:20:38.218540 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index HA_Chassis_Group.name {{(pid=89274) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 14:20:38.218578 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Address_Set.name {{(pid=89274) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 14:20:38.218616 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Connection.target {{(pid=89274) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 14:20:38.218652 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Gateway_Chassis.name {{(pid=89274) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 14:20:38.218688 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Mirror.name {{(pid=89274) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 14:20:38.218724 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Chassis_Template_Var.chassis {{(pid=89274) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 14:20:38.218761 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Logical_Router_Port.name {{(pid=89274) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 14:20:38.218811 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Port_Group.name {{(pid=89274) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 14:20:38.218849 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Logical_Switch_Port.name {{(pid=89274) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 14:20:38.221101 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.4.3.119:6641: connecting... Jun 08 14:20:38.225076 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.4.3.119:6641: connected Jun 08 14:20:38.244930 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Getting OvsdbSbOvnIdl for MaintenanceWorker with retry Jun 08 14:20:38.263761 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Chassis.name {{(pid=89274) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 08 14:20:38.263847 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Port_Binding.logical_port {{(pid=89274) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 08 14:20:38.264026 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Chassis_Private.name {{(pid=89274) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 14:20:38.264026 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Datapath_Binding.tunnel_key {{(pid=89274) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 14:20:38.266189 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.4.3.119:6642: connecting... Jun 08 14:20:38.270149 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.4.3.119:6642: connected Jun 08 14:20:38.298243 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip precommit_update 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.298368 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip precommit_delete 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.298454 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_create 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.298512 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_update 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.298582 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_update 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.298658 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port precommit_delete 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.298921 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.services.portforwarding.drivers.ovn.driver.OVNPortForwarding.register-13435'] for port_forwarding_plugin (None), after_init {{(pid=89274) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Jun 08 14:20:38.298983 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_create 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.299061 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > router_interface after_create 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.299169 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_update 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.299233 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_delete 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.299299 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > router_interface after_delete 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.299435 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [-] Starting OVNClientQosExtension Jun 08 14:20:38.299526 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.placement [-] Starting OVNClientPlacementExtension Jun 08 14:20:38.299919 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment before_delete 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.300003 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet precommit_delete_associations 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.300329 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_create 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.300417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_delete 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.300489 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_create 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.300546 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_delete 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.300607 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_update 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.300670 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_delete 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.300791 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_create 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.300791 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.300870 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_update 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.364230 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_create 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.364337 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.364419 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet before_create 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.364496 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet before_update 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.368021 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group after_delete 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.368476 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_delete 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.368476 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.services.logapi.drivers.ovn.driver.DriverBase._register-3979627'] for logging-plugin (None), after_init {{(pid=89274) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Jun 08 14:20:38.368476 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_create 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.368573 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_update 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:38.369219 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_delete 55550000, False {{(pid=89274) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 14:20:47.892368 np0000002315 neutron-ovn-maintenance-worker[88443]: DEBUG dbcounter [-] [88443] Writing DB stats neutron:SELECT=4 {{(pid=88443) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:20:48.369198 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [-] OVN-Northbound DB sync process started @ 2026-06-08 14:20:48.368994 {{(pid=89274) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:99}} Jun 08 14:20:48.370398 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync port groups started @ 2026-06-08 14:20:48.370120 {{(pid=89274) sync_port_groups /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:132}} Jun 08 14:20:48.415264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writer thread running {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:102}} Jun 08 14:20:48.430725 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync port groups completed @ 2026-06-08 14:20:48.430504 {{(pid=89274) sync_port_groups /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:196}} Jun 08 14:20:48.430861 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync networks, ports and DHCP options started @ 2026-06-08 14:20:48.430752 {{(pid=89274) sync_networks_ports_and_dhcp_opts /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1152}} Jun 08 14:20:48.710588 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.services.segments.db [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] neutron.services.segments.plugin.Plugin method get_segments called with arguments (,) {'filters': {'network_id': ['fffd4e9d-b343-4a97-8e5f-4015f805fc46'], 'is_dynamic': False}} {{(pid=89274) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} Jun 08 14:20:48.714947 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.services.segments.db [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] neutron.services.segments.plugin.Plugin method get_segments called with arguments (,) {'filters': {'network_id': ['a137566a-eaf1-4f5d-a192-15d63e6c049a'], 'is_dynamic': False}} {{(pid=89274) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} Jun 08 14:20:48.718787 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync metadata ports started {{(pid=89274) _sync_metadata_ports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1097}} Jun 08 14:20:48.781017 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync metadata ports completed {{(pid=89274) _sync_metadata_ports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1149}} Jun 08 14:20:48.781137 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync DHCP options for Neutron subnets started {{(pid=89274) _sync_subnet_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:949}} Jun 08 14:20:48.831505 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync DHCP options for Neutron subnets completed @ 2026-06-08 14:20:48.831209 {{(pid=89274) _sync_subnet_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1022}} Jun 08 14:20:48.832065 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 08 14:20:48.832371 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync DHCP options for Neutron ports with extra dhcp options assigned started {{(pid=89274) _sync_port_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1027}} Jun 08 14:20:48.832496 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync DHCP options for Neutron ports with extra dhcp options assigned completed {{(pid=89274) _sync_port_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1086}} Jun 08 14:20:48.832624 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync networks, ports and DHCP options completed @ 2026-06-08 14:20:48.832493 {{(pid=89274) sync_networks_ports_and_dhcp_opts /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1321}} Jun 08 14:20:48.832775 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync ACLs started @ 2026-06-08 14:20:48.832655 {{(pid=89274) sync_acls /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:239}} Jun 08 14:20:48.867572 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync ACLs completed @ 2026-06-08 14:20:48.867367 {{(pid=89274) sync_acls /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:431}} Jun 08 14:20:48.867682 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync routers and router ports started @ 2026-06-08 14:20:48.867584 {{(pid=89274) sync_routers_and_rports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:736}} Jun 08 14:20:49.083367 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 08 14:20:49.083618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync routers and router ports completed @ 2026-06-08 14:20:49.083373 {{(pid=89274) sync_routers_and_rports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:944}} Jun 08 14:20:49.083757 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync port QoS policies started @ 2026-06-08 14:20:49.083628 {{(pid=89274) sync_port_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1369}} Jun 08 14:20:49.083924 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] Starting OVNClientQosExtension Jun 08 14:20:49.098769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync port QoS policies completed @ 2026-06-08 14:20:49.098560 {{(pid=89274) sync_port_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1392}} Jun 08 14:20:49.098893 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync Floating IP QoS policies started @ 2026-06-08 14:20:49.098759 {{(pid=89274) sync_fip_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1397}} Jun 08 14:20:49.098984 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] Starting OVNClientQosExtension Jun 08 14:20:49.119750 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync Floating IP QoS policies completed @ 2026-06-08 14:20:49.119528 {{(pid=89274) sync_fip_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1420}} Jun 08 14:20:49.119905 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync Floating IP NAT rules started @ 2026-06-08 14:20:49.119747 {{(pid=89274) sync_fip_dnat_rules /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1425}} Jun 08 14:20:49.120134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-NB Sync Floating IP NAT rules completed @ 2026-06-08 14:20:49.120003 {{(pid=89274) sync_fip_dnat_rules /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1451}} Jun 08 14:20:49.120192 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-fcb3e5ba-17bd-41b5-9b74-0b729d8f714e None None] OVN-Northbound DB sync process completed @ 2026-06-08 14:20:49.120116 {{(pid=89274) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:112}} Jun 08 14:20:58.385930 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [-] OVN-Southbound DB sync process started @ 2026-06-08 14:20:58.385237 {{(pid=89274) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1480}} Jun 08 14:20:58.386615 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [-] Starting OVNClientQosExtension Jun 08 14:20:58.387045 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-57b73474-4ad0-45b0-a5b3-182a4699979b None None] OVN-SB Sync hostname and physical networks started @ 2026-06-08 14:20:58.386771 {{(pid=89274) sync_hostname_and_physical_networks /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1511}} Jun 08 14:20:58.389701 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.389913 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.390044 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_dhcp_options_consistency {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.390148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_fair_meter_consistency {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.390246 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_fdb_aging_settings {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.390388 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_for_igmp_snoop_support {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.390477 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_for_inconsistencies {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.390985 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_global_dhcp_opts {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.391148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.391242 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.391370 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_provider_distributed_ports {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.391452 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.391543 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.391620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.cleanup_old_hash_ring_nodes {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.391714 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.configure_nb_global {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.391860 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.391978 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_fip_distributed_flag {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.392100 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_network_type_and_physnet {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.392193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_ovn_owned_dns_option {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.392311 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_ha_failover {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.392414 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.392503 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_mac_aging_settings {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.392661 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_router_distributed_flag {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.392739 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_security_group_with_address_group {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.392902 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_virtual_port_parent_hostname {{(pid=89274) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 14:20:58.394269 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-57b73474-4ad0-45b0-a5b3-182a4699979b None None] Agents populated in the agent cache: {{(pid=89274) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:306}} Jun 08 14:20:58.394451 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-57b73474-4ad0-45b0-a5b3-182a4699979b None None] - Agent type: OVN Controller Gateway agent, hostname: np0000002315, name: 8011067c-eb29-42f2-be2f-efa81d584fdd {{(pid=89274) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:308}} Jun 08 14:20:58.394844 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-57b73474-4ad0-45b0-a5b3-182a4699979b None None] - Agent type: OVN Neutron agent, hostname: np0000002315, name: 8011067c-eb29-42f2-be2f-efa81d584fdd {{(pid=89274) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:308}} Jun 08 14:20:58.396093 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.396278 np0000002315 neutron-ovn-maintenance-worker[89274]: WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-57b73474-4ad0-45b0-a5b3-182a4699979b None None] New host np0000002315 found in OVN SB DB, but not in Neutron. Add its SegmentHostMapping in Neutron Jun 08 14:20:58.396627 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [-] Maintenance task thread has started Jun 08 14:20:58.397629 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.399498 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_dhcp_options_consistency' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.403648 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [-] MaintenanceWorker process has finished the post initialization Jun 08 14:20:58.404855 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fair_meter_consistency' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.406136 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): ovsdb-client transact ssl:10.4.3.119:6641 --timeout 180 -p /opt/stack/data/CA/int-ca/private/devstack-cert.key -c /opt/stack/data/CA/int-ca/devstack-cert.crt -C /opt/stack/data/CA/int-ca/ca-chain.pem ["OVN_Northbound", {"op": "wait", "timeout": 0, "table": "Port_Group", "where": [["name", "==", "pvlan_pg_drop"]], "until": "==", "rows": []}, {"op": "insert", "table": "ACL", "row": {"action": "drop", "direction": "to-lport", "match": "outport == @pvlan_pg_drop && ip", "priority": 1010}, "uuid-name": "pvlandroptoport"}, {"op": "insert", "table": "ACL", "row": {"action": "drop", "direction": "from-lport", "match": "inport == @pvlan_pg_drop && ip", "priority": 1010}, "uuid-name": "pvlandropfromport"}, {"op": "insert", "table": "Port_Group", "row": {"name": "pvlan_pg_drop", "acls": ["set", [["named-uuid", "pvlandroptoport"], ["named-uuid", "pvlandropfromport"]]]}}] {{(pid=89274) execute /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:440}} Jun 08 14:20:58.406348 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.406508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_igmp_snoop_support' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.406608 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.406694 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_global_dhcp_opts' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.406786 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.406883 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.406984 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_provider_distributed_ports' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.407073 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.407159 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.407463 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.cleanup_old_hash_ring_nodes' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.407546 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.configure_nb_global' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.407642 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.407722 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.407820 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_network_type_and_physnet' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.407920 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_ovn_owned_dns_option' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.407996 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.408078 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.408166 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.408275 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_distributed_flag' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.409133 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_security_group_with_address_group' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.410374 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_virtual_port_parent_hostname' {{(pid=89274) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 14:20:58.425194 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG oslo_concurrency.lockutils [None req-57b73474-4ad0-45b0-a5b3-182a4699979b None None] Acquiring lock "update_segment_host_mapping" by "neutron.services.segments.db.update_segment_host_mapping" {{(pid=89274) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:511}} Jun 08 14:20:58.425468 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG oslo_concurrency.lockutils [None req-57b73474-4ad0-45b0-a5b3-182a4699979b None None] Lock "update_segment_host_mapping" acquired by "neutron.services.segments.db.update_segment_host_mapping" :: waited 0.001s {{(pid=89274) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:524}} Jun 08 14:20:58.436825 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG oslo_concurrency.processutils [-] CMD "ovsdb-client transact ssl:10.4.3.119:6641 --timeout 180 -p /opt/stack/data/CA/int-ca/private/devstack-cert.key -c /opt/stack/data/CA/int-ca/devstack-cert.crt -C /opt/stack/data/CA/int-ca/ca-chain.pem ["OVN_Northbound", {"op": "wait", "timeout": 0, "table": "Port_Group", "where": [["name", "==", "pvlan_pg_drop"]], "until": "==", "rows": []}, {"op": "insert", "table": "ACL", "row": {"action": "drop", "direction": "to-lport", "match": "outport == @pvlan_pg_drop && ip", "priority": 1010}, "uuid-name": "pvlandroptoport"}, {"op": "insert", "table": "ACL", "row": {"action": "drop", "direction": "from-lport", "match": "inport == @pvlan_pg_drop && ip", "priority": 1010}, "uuid-name": "pvlandropfromport"}, {"op": "insert", "table": "Port_Group", "row": {"name": "pvlan_pg_drop", "acls": ["set", [["named-uuid", "pvlandroptoport"], ["named-uuid", "pvlandropfromport"]]]}}]" returned: 0 in 0.033s {{(pid=89274) execute /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:468}} Jun 08 14:20:58.445227 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron_lib.callbacks.manager [None req-57b73474-4ad0-45b0-a5b3-182a4699979b None None] Publish callbacks ['neutron.plugins.ml2.plugin.DhcpAgentSchedulerDbMixin.auto_schedule_new_network_segments-1943639', 'neutron.services.segments.plugin.NovaSegmentNotifier._notify_host_addition_to_aggregate-577121'] for segment_host_mapping (None), after_create {{(pid=89274) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Jun 08 14:20:58.457175 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.services.segments.db [None req-57b73474-4ad0-45b0-a5b3-182a4699979b None None] Segments {'0817ce61-1c85-4457-911a-0b8fdccf389d'} mapped to the host np0000002315 {{(pid=89274) update_segment_host_mapping /opt/stack/neutron/neutron/services/segments/db.py:249}} Jun 08 14:20:58.461603 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG oslo_concurrency.lockutils [None req-57b73474-4ad0-45b0-a5b3-182a4699979b None None] Lock "update_segment_host_mapping" "released" by "neutron.services.segments.db.update_segment_host_mapping" :: held 0.036s {{(pid=89274) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:543}} Jun 08 14:20:58.462480 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-57b73474-4ad0-45b0-a5b3-182a4699979b None None] OVN-SB Sync hostname and physical networks completed @ 2026-06-08 14:20:58.462300 {{(pid=89274) sync_hostname_and_physical_networks /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1538}} Jun 08 14:20:58.465421 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89274) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 14:20:58.465624 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_baremetal_ports_dhcp_options {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:58.465743 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Sync baremetal port DHCP options with configuration. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:58.519500 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_baremetal_ports_dhcp_options finished in 0.053 seconds Jun 08 14:20:58.519731 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89274) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 14:20:58.520146 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_dhcp_options_consistency {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:58.520262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Backfill network_id on DHCP_Options external_ids. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:58.573752 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_dhcp_options_consistency finished in 0.053 seconds Jun 08 14:20:58.574678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_dhcp_options_consistency' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89274) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 14:20:58.574837 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fair_meter_consistency {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:58.574938 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Refresh ACL logging fair meter after configuration reload. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:58.575214 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fair_meter_consistency finished in 0.000 seconds Jun 08 14:20:58.575372 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fair_meter_consistency' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89274) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 14:20:58.575523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:58.575639 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:58.578194 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:20:58.842282 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:20:58.842524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:58.846008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:20:58.846008 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:58.846008 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:20:58.846008 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:20:58.846008 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:58.846008 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:20:58.846008 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:58.846008 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:20:58.851226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.270 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:58.851226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:20:58.851226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:20:58.851226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:20:58.851226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:20:58.851226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:20:58.851226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:20:58.851226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:58.851226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:20:58.851226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:20:58.851226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:20:58.851226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:58.851598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:20:58.851598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:20:58.851598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:20:58.851598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:20:58.851598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:20:58.851598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:20:58.851598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:20:58.851598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:20:58.851598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:58.851598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:20:58.851598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:20:58.851598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:58.851598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:20:58.851813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:58.851813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:20:58.854682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:58.854682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:20:58.854682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:20:58.854682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:20:58.854682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:20:58.854682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:20:58.854682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:20:58.854682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:20:58.854682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:20:58.854682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:20:58.854682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:20:58.854682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:20:58.854682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:20:58.854682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:20:58.854682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:20:58.854682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:20:58.855066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:20:58.855066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:20:58.855066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:58.855066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:20:58.855066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:20:58.855066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:20:58.855066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:58.855066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:20:58.855066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:20:58.855066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:20:58.855066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:20:58.855066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:20:58.855066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:20:58.855066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:20:58.855066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:20:58.855066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:58.855066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:20:58.855066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:20:58.855066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:58.855520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:20:58.855520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:58.855520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:20:58.855520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_for_igmp_snoop_support {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:58.855520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Check for IGMP snoop support. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:58.855658 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_for_igmp_snoop_support finished in 0.000 seconds Jun 08 14:20:58.855819 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_igmp_snoop_support' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89274) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 14:20:58.855973 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:58.856077 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:58.870619 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: No inconsistencies found. Skipping {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:437}} Jun 08 14:20:58.870749 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_for_inconsistencies finished in 0.015 seconds Jun 08 14:20:58.870989 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_global_dhcp_opts {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:58.871264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Check global DHCP options consistency. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:58.871464 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_global_dhcp_opts finished in 0.000 seconds Jun 08 14:20:58.871603 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_global_dhcp_opts' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89274) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 14:20:58.871750 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_localnet_port_has_learn_fdb {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:58.872293 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Ensure localnet ports have learn-fdb set per configuration. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:58.876164 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 08 14:20:58.876444 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_localnet_port_has_learn_fdb finished in 0.002 seconds Jun 08 14:20:58.876602 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89274) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 14:20:58.876752 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_network_broadcast_arps_to_all_routers {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:58.876850 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Sync broadcast-arps-to-all-routers on external networks. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:58.920610 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_network_broadcast_arps_to_all_routers finished in 0.044 seconds Jun 08 14:20:58.920769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89274) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 14:20:58.920946 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_provider_distributed_ports {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:58.921055 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Align reside-on-redirect-chassis on VLAN/FLAT distributed ports. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:59.027727 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 08 14:20:59.028660 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_provider_distributed_ports finished in 0.107 seconds Jun 08 14:20:59.028946 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_provider_distributed_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89274) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 14:20:59.031206 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-57b73474-4ad0-45b0-a5b3-182a4699979b None None] OVN-Southbound DB sync process completed @ 2026-06-08 14:20:59.028978 {{(pid=89274) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1488}} Jun 08 14:20:59.031316 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_redirect_type_router_gateway_ports {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:59.031439 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Align router gateway redirect-type for provider VLAN/FLAT networks. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:59.094540 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_redirect_type_router_gateway_ports finished in 0.063 seconds Jun 08 14:20:59.094740 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89274) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 14:20:59.094935 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:59.095038 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:59.096212 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:20:59.096432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.096922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:20:59.096922 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:59.096922 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:20:59.096922 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:20:59.096922 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:59.096922 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:20:59.096922 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.096922 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:20:59.097944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.097944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:20:59.097944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:20:59.097944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:20:59.097944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.097944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:20:59.097944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:20:59.097944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.097944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:20:59.097944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:20:59.097944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:20:59.097944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.098135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:20:59.098135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:20:59.098135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:20:59.098135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:20:59.098135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:20:59.098135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:20:59.098135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:20:59.098135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:20:59.098135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:59.098135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:20:59.098135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:20:59.098135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:59.098135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:20:59.098436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.098436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:20:59.099016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.099016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:20:59.099016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:20:59.099016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:20:59.099016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:20:59.099016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:20:59.099016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.099016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:20:59.099016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:20:59.099016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.099016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:20:59.099016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:20:59.099016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.099016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:20:59.099016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:20:59.099016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.099311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:20:59.099311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:20:59.099311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.099311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:20:59.099311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:20:59.099311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:20:59.099311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.099311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:20:59.099311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:20:59.099311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:20:59.099311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:20:59.099311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:20:59.099311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:20:59.099311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:20:59.099311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:20:59.099311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:59.099311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:20:59.099311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:20:59.099311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:59.099715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:20:59.099715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.099715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:20:59.099715 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: cleanup_old_hash_ring_nodes {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:59.099715 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Purge stale hash ring nodes older than five days. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:59.103613 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.db.ovn_hash_ring_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Cleaned up Hash Ring nodes older than 5 days Jun 08 14:20:59.103757 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task cleanup_old_hash_ring_nodes finished in 0.004 seconds Jun 08 14:20:59.103965 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: configure_nb_global {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:59.104139 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply ovn_nb_global settings to NB_Global options. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:59.104340 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'ignore_lsp_down': 'false', 'fdb_removal_limit': '0', 'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:20:59.104884 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'ignore_lsp_down': 'false', 'fdb_removal_limit': '0', 'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:20:59.105115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.105584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:20:59.105584 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:59.105584 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:20:59.105584 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:20:59.105584 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:59.105584 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:20:59.105584 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.105584 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:20:59.106902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.106902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:20:59.106902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:20:59.106902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:20:59.106902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.106902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:20:59.106902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:20:59.106902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:20:59.106902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.106902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:20:59.106902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:20:59.106902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:20:59.106902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:20:59.106902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:20:59.106902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:20:59.107145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:20:59.107145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:20:59.107145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:59.107145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:20:59.107145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:20:59.107145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:59.107145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:20:59.107145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.107145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:20:59.108076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task configure_nb_global failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.108076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:20:59.108076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:20:59.108076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:20:59.108076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.108076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1038, in configure_nb_global Jun 08 14:20:59.108076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:20:59.108076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:20:59.108076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:20:59.108076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.108076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:20:59.108324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:20:59.108324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:20:59.108324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.108324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:20:59.108324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:20:59.108324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:20:59.108324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:20:59.108324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:20:59.108324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:20:59.108324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:20:59.108324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:20:59.108324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:59.108324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:20:59.108324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:20:59.108657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:59.108657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:20:59.108657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.108657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:20:59.109135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.configure_nb_global' (it runs every 86400.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.109135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:20:59.109135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:20:59.109135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:20:59.109135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:20:59.109135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:20:59.109135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.109135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:20:59.109135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:20:59.109135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.109135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:20:59.109135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:20:59.109135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.109135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:20:59.109135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:20:59.109135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.109403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1038, in configure_nb_global Jun 08 14:20:59.109403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:20:59.109403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:20:59.109403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:20:59.109403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.109403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:20:59.109403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:20:59.109403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:20:59.109403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.109403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:20:59.109403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:20:59.109403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:20:59.109403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:20:59.109403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:20:59.109403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:20:59.109403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:20:59.109403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:20:59.109403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:59.109403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:20:59.109815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:20:59.109815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:59.109815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:20:59.109815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.109815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:20:59.109815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 08 14:20:59.110049 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89274) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 14:20:59.110219 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:59.110339 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:59.110672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:20:59.111039 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:20:59.111207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.111705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:20:59.111705 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:59.111705 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:20:59.111705 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:20:59.111705 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:59.111705 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:20:59.111705 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.111705 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:20:59.112533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.112533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:20:59.112533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:20:59.112533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:20:59.112533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.112533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:20:59.112533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:20:59.112533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:20:59.112533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.112533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:20:59.112533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:20:59.112533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:20:59.112533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:20:59.112533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:20:59.112533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:20:59.112763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:20:59.112763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:20:59.112763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:59.112763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:20:59.112763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:20:59.112763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:59.112763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:20:59.112763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.112763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:20:59.113395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.113395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:20:59.113395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:20:59.113395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:20:59.113395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.113395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:20:59.113395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:20:59.113395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:20:59.113395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:20:59.113395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:20:59.113395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.113395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:20:59.113592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:20:59.113592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:20:59.113592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.113592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:20:59.113592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:20:59.113592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:20:59.113592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:20:59.113592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:20:59.113592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:20:59.113592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:20:59.113592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:20:59.113592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:59.113592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:20:59.113592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:20:59.113913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:59.113913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:20:59.113913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.113913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:20:59.114463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.114463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:20:59.114463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:20:59.114463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:20:59.114463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:20:59.114463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:20:59.114463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.114463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:20:59.114463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:20:59.114463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.114463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:20:59.114463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:20:59.114463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.114463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:20:59.114463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:20:59.114463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.114694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:20:59.114694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:20:59.114694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:20:59.114694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:20:59.114694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:20:59.114694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.114694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:20:59.114694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:20:59.114694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:20:59.114694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.114694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:20:59.114694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:20:59.114694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:20:59.114694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:20:59.114694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:20:59.114694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:20:59.114694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:20:59.114694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:20:59.115012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:59.115012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:20:59.115012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:20:59.115012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:59.115012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:20:59.115012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.115012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:20:59.115012 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_network_type_and_physnet {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:59.115012 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set network type and physnet external_ids on logical switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:59.120651 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_network_type_and_physnet finished in 0.006 seconds Jun 08 14:20:59.120786 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_network_type_and_physnet' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89274) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 14:20:59.120972 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_ovn_owned_dns_option {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:59.121090 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply ovn_owned option on DNS records per configuration. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:59.121316 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_ovn_owned_dns_option finished in 0.000 seconds Jun 08 14:20:59.121447 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_ovn_owned_dns_option' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89274) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 14:20:59.121592 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:59.121762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:59.122061 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:20:59.122369 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:20:59.122569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.122959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:20:59.122959 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:59.122959 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:20:59.122959 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:20:59.122959 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:59.122959 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:20:59.122959 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.122959 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:20:59.123564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.123564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:20:59.123564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:20:59.123564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:20:59.123564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.123564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:20:59.123564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:20:59.123564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:20:59.123564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.123564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:20:59.123564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:20:59.123564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:20:59.123564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:20:59.123564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:20:59.123564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:20:59.123824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:20:59.123824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:20:59.123824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:59.123824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:20:59.123824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:20:59.123824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:59.123824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:20:59.123824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.123824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:20:59.124361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.124361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:20:59.124361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:20:59.124361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:20:59.124361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.124361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:20:59.124361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:20:59.124361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:20:59.124361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:20:59.124361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.124361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:20:59.124570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:20:59.124570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:20:59.124570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.124570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:20:59.124570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:20:59.124570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:20:59.124570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:20:59.124570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:20:59.124570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:20:59.124570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:20:59.124570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:20:59.124570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:59.124570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:20:59.124570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:20:59.124795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:59.124795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:20:59.124795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.124795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:20:59.125288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.125288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:20:59.125288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:20:59.125288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:20:59.125288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:20:59.125288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:20:59.125288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.125288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:20:59.125288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:20:59.125288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.125288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:20:59.125288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:20:59.125288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.125288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:20:59.125288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:20:59.125288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.125288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:20:59.125572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:20:59.125572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:20:59.125572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:20:59.125572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.125572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:20:59.125572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:20:59.125572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:20:59.125572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.125572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:20:59.125572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:20:59.125572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:20:59.125572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:20:59.125572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:20:59.125572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:20:59.125572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:20:59.125572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:20:59.125572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:59.125572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:20:59.125572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:20:59.125855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:59.125855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:20:59.125855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.125855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:20:59.125855 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_lrouter_ports_ext_ids_name_prefix {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:59.125855 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: ('Check missing prefix router_name', 'in external_ids of LRPs.') {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:59.126386 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_lrouter_ports_ext_ids_name_prefix finished in 0.000 seconds Jun 08 14:20:59.126515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89274) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 14:20:59.126649 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:59.126756 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:59.126927 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:20:59.127283 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:20:59.127472 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:20:59.127577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.127891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:20:59.127891 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:59.127891 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:20:59.127891 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:20:59.127891 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:59.127891 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:20:59.127891 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.127891 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:20:59.128479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.128479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:20:59.128479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:20:59.128479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:20:59.128479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.128479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:20:59.128479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:20:59.128479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.128479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:20:59.128479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:20:59.128479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:20:59.128479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.128682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:20:59.128682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:20:59.128682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:20:59.128682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:20:59.128682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:20:59.128682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:20:59.128682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:20:59.128682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:20:59.128682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:59.128682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:20:59.128682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:20:59.128682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:59.128682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:20:59.128889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.128889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:20:59.129310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.129310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:20:59.129310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:20:59.129310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:20:59.129310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:20:59.129310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:20:59.129310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.129310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:20:59.129310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:20:59.129310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.129310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:20:59.129310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:20:59.129310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.129310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:20:59.129310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:20:59.129310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:20:59.129567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:20:59.129567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:20:59.129567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.129567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:20:59.129567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:20:59.129567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:20:59.129567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:20:59.129567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:20:59.129567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:20:59.129567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:20:59.129567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:20:59.129567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:20:59.129567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:20:59.129567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:20:59.129567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:20:59.129567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:20:59.129567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:20:59.129567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:20:59.129567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:20:59.129854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:20:59.129854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:20:59.129854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:20:59.129854 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_router_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:59.129854 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Sync router.distributed with distributed floating IP config. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:59.132680 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_router_distributed_flag finished in 0.003 seconds Jun 08 14:20:59.132822 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_distributed_flag' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89274) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 14:20:59.132991 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_security_group_with_address_group {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:20:59.133118 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Repair Address_Sets and ACLs for address group rules. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:20:59.143454 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_security_group_with_address_group finished in 0.010 seconds Jun 08 14:20:59.143612 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_security_group_with_address_group' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89274) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 14:20:59.144501 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_virtual_port_parent_hostname' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89274) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 14:21:03.575718 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:03.576171 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:03.576238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:03.576932 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:03.602706 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:03.602881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:03.603344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:03.603344 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:03.603344 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:03.603344 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:03.603344 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:03.603344 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:03.603344 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:03.603344 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:03.603949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:03.603949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:03.603949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:03.603949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:03.603949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:03.603949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:03.603949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:03.603949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:03.603949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:03.603949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:03.603949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:03.603949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:03.604195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:03.604195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:03.604195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:03.604195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:03.604195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:03.604195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:03.604195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:03.604195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:03.604195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:03.604195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:03.604195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:03.604195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:03.604195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:03.604628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:03.604628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:03.604739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:03.604739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:03.604739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:03.604739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:03.604739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:03.604739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:03.604739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:03.604739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:03.604739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:03.604739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:03.604739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:03.604739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:03.604739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:03.604739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:03.604739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:03.604739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:03.605044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:03.605044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:03.605044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:03.605044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:03.605044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:03.605044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:03.605044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:03.605044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:03.605044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:03.605044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:03.605044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:03.605044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:03.605044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:03.605044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:03.605044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:03.605044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:03.605044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:03.605044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:03.605044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:03.605376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:03.605376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:03.605376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:04.095101 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:04.095351 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:04.095469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:04.095902 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:04.096057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.096524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:04.096524 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:04.096524 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:04.096524 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:04.096524 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:04.096524 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:04.096524 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.096524 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:04.097176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.097176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:04.097176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:04.097176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:04.097176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.097176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:04.097176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:04.097176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.097176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:04.097176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:04.097176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:04.097176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.097500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:04.097500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:04.097500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:04.097500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:04.097500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:04.097500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:04.097500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:04.097500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:04.097500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:04.097500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:04.097500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:04.097500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:04.097500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:04.097719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.097719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:04.097841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.097841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:04.097841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:04.097841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:04.097841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:04.097841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:04.097841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.097841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:04.097841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:04.097841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.097841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:04.097841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:04.097841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.097841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:04.097841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:04.097841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.098159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:04.098159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:04.098159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.098159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:04.098159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:04.098159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:04.098159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.098159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:04.098159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:04.098159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:04.098159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:04.098159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:04.098159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:04.098159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:04.098159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:04.098159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:04.098159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:04.098159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:04.098159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:04.098506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:04.098506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.098506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:04.110371 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:04.110592 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:04.110678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:04.110773 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:21:04.111077 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:04.111264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.114947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:04.114947 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:04.114947 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:04.114947 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:04.114947 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:04.114947 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:04.114947 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.114947 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:04.115501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.115501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:04.115501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:04.115501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:04.115501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.115501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:04.115501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:04.115501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:04.115501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.115501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:04.115501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:04.115501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:04.115501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:04.115501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:04.115501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:04.115778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:04.115778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:04.115778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:04.115778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:04.115778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:04.115778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:04.115778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:04.115778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.115778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:04.116067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.116067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:04.116067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:04.116067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:04.116067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.116067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:04.116067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:04.116067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:21:04.116067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:04.116067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:04.116067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.116067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:04.116280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:04.116280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:04.116280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.116280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:04.116280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:04.116280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:04.116280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:04.116280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:04.116280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:04.116280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:04.116280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:04.116280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:04.116280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:04.116280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:04.116496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:04.116496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:04.116496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.116496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:04.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:04.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:04.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:04.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:04.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:04.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:04.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:04.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:04.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:04.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:04.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:04.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.117064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:04.117064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:04.117064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:21:04.117064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:04.117064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:04.117064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.117064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:04.117064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:04.117064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:04.117064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.117064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:04.117064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:04.117064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:04.117064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:04.117064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:04.117064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:04.117064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:04.117064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:04.117360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:04.117360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:04.117360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:04.117360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:04.117360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:04.117360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.117360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:04.121646 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:04.121826 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:04.122993 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:04.122993 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:04.122993 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:04.122993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.123098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:04.123098 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:04.123098 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:04.123098 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:04.123098 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:04.123098 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:04.123098 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.123098 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:04.124297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.124297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:04.124297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:04.124297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:04.124297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.124297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:04.124297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:04.124297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:04.124297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.124297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:04.124297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:04.124297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:04.124297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:04.124297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:04.124297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:04.124715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:04.124715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:04.124715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:04.124715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:04.124715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:04.124715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:04.124715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:04.124715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.124715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:04.124715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.124715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:04.124715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:04.124715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:04.124715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:04.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:04.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:04.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:04.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:04.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:04.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:04.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:04.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:04.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:04.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:04.125293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:04.125293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:04.125293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:04.125293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:04.125293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:04.125293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:04.125293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:04.125293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:04.125293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:04.125293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.125293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:04.125293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.125550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:04.125550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:04.125550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:04.125550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:04.125550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:04.125550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.125550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:04.125550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:04.125550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.125550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:04.125550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:04.125550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.125550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:04.125550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:04.125550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.125550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:04.125550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:04.125550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:04.125550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.125924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:04.127129 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:04.127129 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:04.127129 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:04.127275 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:04.127515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:04.127730 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:04.127851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.128276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:04.128276 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:04.128276 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:04.128276 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:04.128276 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:04.128276 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:04.128276 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.128276 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:04.128809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.128809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:04.128809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:04.128809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:04.128809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.128809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:04.128809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:04.128809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.128809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:04.128809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:04.128809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:04.128809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.129074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:04.129074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:04.129074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:04.129074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:04.129074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:04.129074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:04.129074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:04.129074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:04.129074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:04.129074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:04.129074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:04.129074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:04.129074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:04.129351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.129351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:04.129563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.129563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:04.129563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:04.129563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:04.129563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:04.129563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:04.129563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.129563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:04.129563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:04.129563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.129563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:04.129563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:04.129563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.129563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:04.129563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:04.129563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:04.129845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:04.129845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:04.129845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.129845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:04.129845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:04.129845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:04.129845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:04.129845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:04.129845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:04.129845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:04.129845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:04.129845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:04.129845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:04.129845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:04.129845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:04.129845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:04.129845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:04.129845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:04.129845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:04.130169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:04.130169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:04.130169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:08.576600 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:08.576998 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:08.576998 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:08.577834 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:08.611066 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:08.611559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:08.612211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:08.612211 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:08.612211 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:08.612211 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:08.612211 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:08.612211 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:08.612211 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:08.612211 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:08.613297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:08.613297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:08.613297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:08.613297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:08.613297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:08.613297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:08.613297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:08.613297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:08.613297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:08.613297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:08.613297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:08.613297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:08.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:08.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:08.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:08.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:08.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:08.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:08.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:08.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:08.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:08.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:08.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:08.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:08.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:08.614125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:08.614125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:08.614473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:08.614473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:08.614473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:08.614473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:08.614473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:08.614473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:08.614473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:08.614473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:08.614473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:08.614473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:08.614473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:08.614473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:08.614473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:08.614473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:08.614473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:08.614473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:08.615062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:08.615062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:08.615062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:08.615062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:08.615062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:08.615062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:08.615062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:08.615062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:08.615062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:08.615062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:08.615062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:08.615062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:08.615062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:08.615062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:08.615062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:08.615062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:08.615062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:08.615062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:08.615062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:08.615789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:08.615789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:08.615789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:09.095672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:09.096233 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:09.096378 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:09.099628 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:09.099792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.100866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:09.100866 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:09.100866 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:09.100866 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:09.100866 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:09.100866 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:09.100866 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.100866 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:09.102117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.102117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:09.102117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:09.102117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:09.102117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.102117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:09.102117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:09.102117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.102117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:09.102117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:09.102117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:09.102117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.102482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:09.102482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:09.102482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:09.102482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:09.102482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:09.102482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:09.102482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:09.102482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:09.102482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:09.102482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:09.102482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:09.102482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:09.102482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:09.102709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.102709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:09.104798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.104798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:09.104798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:09.104798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:09.104798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:09.104798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:09.104798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.104798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:09.104798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:09.104798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.104798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:09.104798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:09.104798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.104798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:09.104798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:09.104798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.105163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:09.105163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:09.105163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.105163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:09.105163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:09.105163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:09.105163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.105163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:09.105163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:09.105163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:09.105163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:09.105163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:09.105163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:09.105163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:09.105163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:09.105163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:09.105163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:09.105163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:09.105163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:09.105510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:09.105510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.105510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:09.110793 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:09.111420 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:09.111420 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:09.111755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:21:09.112931 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:09.113160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.113807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:09.113807 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:09.113807 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:09.113807 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:09.113807 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:09.113807 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:09.113807 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.113807 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:09.114695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.114695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:09.114695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:09.114695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:09.114695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.114695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:09.114695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:09.114695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:09.114695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.114695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:09.114695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:09.114695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:09.114695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:09.114695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:09.114695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:09.114986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:09.114986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:09.114986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:09.114986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:09.114986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:09.114986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:09.114986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:09.114986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.114986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:09.115829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.115829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:09.115829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:09.115829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:09.115829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.115829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:09.115829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:09.115829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:21:09.115829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:09.115829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:09.115829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.115829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:09.116072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:09.116072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:09.116072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.116072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:09.116072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:09.116072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:09.116072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:09.116072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:09.116072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:09.116072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:09.116072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:09.116072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:09.116072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:09.116072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:09.116330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:09.116330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:09.116330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.116330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:09.117196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.117196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:09.117196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:09.117196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:09.117196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:09.117196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:09.117196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.117196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:09.117196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:09.117196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.117196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:09.117196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:09.117196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.117196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:09.117196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:09.117196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.117505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:09.117505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:09.117505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:21:09.117505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:09.117505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:09.117505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.117505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:09.117505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:09.117505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:09.117505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.117505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:09.117505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:09.117505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:09.117505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:09.117505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:09.117505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:09.117505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:09.117505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:09.117795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:09.117795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:09.117795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:09.117795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:09.117795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:09.117795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.117795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:09.122035 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:09.122396 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:09.122580 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:09.122784 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:09.123689 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:09.123689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.123928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:09.123928 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:09.123928 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:09.123928 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:09.123928 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:09.123928 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:09.123928 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.123928 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:09.124967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.124967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:09.124967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:09.124967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:09.124967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.124967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:09.124967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:09.124967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:09.124967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.124967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:09.124967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:09.124967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:09.124967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:09.124967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:09.124967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:09.125232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:09.125232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:09.125232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:09.125232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:09.125232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:09.125232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:09.125232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:09.125232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.125232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:09.126355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.126355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:09.126355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:09.126355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:09.126355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.126355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:09.126355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:09.126355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:09.126355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:09.126355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.126355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:09.126539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:09.126539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:09.126539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.126539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:09.126539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:09.126539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:09.126539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:09.126539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:09.126539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:09.126539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:09.126539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:09.126539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:09.126539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:09.126539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:09.126739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:09.126739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:09.126739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.126739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:09.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:09.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:09.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:09.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:09.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:09.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:09.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:09.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:09.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:09.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:09.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:09.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:09.128181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:09.128181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:09.128181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:09.128181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.128181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:09.128181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:09.128181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:09.128181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.128181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:09.128181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:09.128181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:09.128181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:09.128181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:09.128181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:09.128181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:09.128181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:09.128181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:09.128181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:09.128181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:09.128489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:09.128489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:09.128489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.128489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:09.128489 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:09.128572 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:09.128921 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:09.129142 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:09.129959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:09.129959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:09.129959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.130233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:09.130233 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:09.130233 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:09.130233 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:09.130233 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:09.130233 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:09.130233 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.130233 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:09.131190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.131190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:09.131190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:09.131190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:09.131190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.131190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:09.131190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:09.131190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.131190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:09.131190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:09.131190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:09.131190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.131442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:09.131442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:09.131442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:09.131442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:09.131442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:09.131442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:09.131442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:09.131442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:09.131442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:09.131442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:09.131442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:09.131442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:09.131442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:09.131699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.131699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:09.132789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.132789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:09.132789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:09.132789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:09.132789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:09.132789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:09.132789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.132789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:09.132789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:09.132789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.132789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:09.132789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:09.132789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.132789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:09.132789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:09.132789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:09.133193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:09.133193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:09.133193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.133193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:09.133193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:09.133193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:09.133193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:09.133193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:09.133193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:09.133193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:09.133193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:09.133193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:09.133193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:09.133193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:09.133193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:09.133193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:09.133193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:09.133193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:09.133193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:09.133530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:09.133530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:09.133530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:13.577044 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:13.577559 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:13.577559 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:13.577977 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:13.602527 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:13.602723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:13.603158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:13.603158 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:13.603158 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:13.603158 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:13.603158 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:13.603158 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:13.603158 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:13.603158 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:13.603950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:13.603950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:13.603950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:13.603950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:13.603950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:13.603950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:13.603950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:13.603950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:13.603950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:13.603950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:13.603950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:13.603950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:13.604684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:13.604684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:13.604684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:13.604684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:13.604684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:13.604684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:13.604684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:13.604684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:13.604684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:13.604684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:13.604684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:13.604684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:13.604684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:13.605166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:13.605166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:13.605166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:13.605166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:13.605166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:13.605166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:13.605166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:13.605166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:13.605166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:13.605166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:13.605166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:13.605166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:13.605166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:13.605166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:13.605166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:13.605608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:13.605608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:13.605608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:13.605608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:13.605608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:13.605608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:13.605608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:13.605608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:13.605608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:13.605608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:13.605608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:13.605608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:13.605608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:13.605608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:13.605608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:13.605608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:13.605608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:13.605608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:13.605608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:13.606057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:13.606057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:13.606057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:13.606057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:13.606057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:13.606057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:14.097145 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:14.097145 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:14.097145 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:14.097635 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:14.097825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.098317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:14.098317 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:14.098317 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:14.098317 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:14.098317 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:14.098317 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:14.098317 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.098317 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:14.099022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.099022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:14.099022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:14.099022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:14.099022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.099022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:14.099022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:14.099022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.099022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:14.099022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:14.099022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:14.099022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.099340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:14.099340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:14.099340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:14.099340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:14.099340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:14.099340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:14.099340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:14.099340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:14.099340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:14.099340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:14.099340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:14.099340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:14.099340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:14.099655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.099655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:14.100057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.100057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:14.100057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:14.100057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:14.100057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:14.100057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:14.100057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.100057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:14.100057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:14.100057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.100057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:14.100057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:14.100057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.100057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:14.100057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:14.100057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.101175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:14.101175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:14.101175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.101175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:14.101175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:14.101175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:14.101175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.101175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:14.101175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:14.101175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:14.101175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:14.101175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:14.101175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:14.101175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:14.101175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:14.101175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:14.101175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:14.101175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:14.101175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:14.101691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:14.101691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.101691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:14.111321 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:14.111608 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:14.111720 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:14.111787 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:21:14.112099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:14.112270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.112679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:14.112679 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:14.112679 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:14.112679 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:14.112679 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:14.112679 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:14.112679 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.112679 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:14.113266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.113266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:14.113266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:14.113266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:14.113266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.113266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:14.113266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:14.113266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:14.113266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.113266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:14.113266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:14.113266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:14.113266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:14.113266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:14.113266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:14.113677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:14.113677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:14.113677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:14.113677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:14.113677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:14.113677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:14.113677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:14.113677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.113677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:14.113917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.113917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:14.113917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:14.113917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:14.113917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.113917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:14.113917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:14.113917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:21:14.113917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:14.113917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:14.113917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.113917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:14.114220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:14.114220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:14.114220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.114220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:14.114220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:14.114220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:14.114220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:14.114220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:14.114220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:14.114220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:14.114220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:14.114220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:14.114220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:14.114220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:14.114561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:14.114561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:14.114561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.114561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:14.114561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.114561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:14.114561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:14.114561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:14.114561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:14.114561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:14.114561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.114561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:14.114561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:14.114561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.114916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:14.114916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:14.114916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.114916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:14.114916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:14.114916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.114916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:14.114916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:14.114916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:21:14.114916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:14.114916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:14.114916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.114916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:14.114916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:14.114916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:14.114916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.114916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:14.114916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:14.114916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:14.115392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:14.115392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:14.115392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:14.115392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:14.115392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:14.115392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:14.115392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:14.115392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:14.115392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:14.115392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:14.115392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.115392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:14.122492 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:14.122741 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:14.122853 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:14.122929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:14.123413 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:14.123663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.124172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:14.124172 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:14.124172 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:14.124172 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:14.124172 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:14.124172 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:14.124172 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.124172 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:14.124785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.124785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:14.124785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:14.124785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:14.124785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.124785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:14.124785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:14.124785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:14.124785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.124785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:14.124785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:14.124785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:14.124785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:14.124785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:14.124785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:14.125165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:14.125165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:14.125165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:14.125165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:14.125165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:14.125165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:14.125165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:14.125165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.125165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:14.125627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.125627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:14.125627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:14.125627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:14.125627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.125627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:14.125627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:14.125627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:14.125627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:14.125627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.125627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:14.126550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:14.126550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:14.126550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.126550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:14.126550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:14.126550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:14.126550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:14.126550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:14.126550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:14.126550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:14.126550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:14.126550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:14.126550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:14.126550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:14.126893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:14.126893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:14.126893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.126893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:14.126893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.126893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:14.126893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:14.126893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:14.126893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:14.126893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:14.126893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.126893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:14.126893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:14.126893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.127235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:14.127235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:14.127235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.127235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:14.127235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:14.127235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.127235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:14.127235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:14.127235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:14.127235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:14.127235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.127235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:14.127235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:14.127235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:14.127235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.127235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:14.127235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:14.127235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:14.127235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:14.127614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:14.127614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:14.127614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:14.127614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:14.127614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:14.127614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:14.127614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:14.127614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:14.127614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:14.127614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.127614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:14.128695 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:14.128889 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:14.128998 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:14.129135 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:14.129476 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:14.129696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:14.129812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.130743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:14.130743 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:14.130743 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:14.130743 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:14.130743 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:14.130743 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:14.130743 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.130743 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:14.131139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.131139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:14.131139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:14.131139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:14.131139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.131139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:14.131139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:14.131139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.131139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:14.131139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:14.131139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:14.131139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.131362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:14.131362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:14.131362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:14.131362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:14.131362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:14.131362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:14.131362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:14.131362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:14.131362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:14.131362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:14.131362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:14.131362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:14.131362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:14.131557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.131557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:14.131858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.131858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:14.131858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:14.131858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:14.131858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:14.131858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:14.131858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.131858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:14.131858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:14.131858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.131858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:14.131858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:14.131858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.131858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:14.131858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:14.131858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:14.132311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:14.132311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:14.132311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.132311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:14.132311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:14.132311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:14.132311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:14.132311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:14.132311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:14.132311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:14.132311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:14.132311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:14.132311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:14.132311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:14.132311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:14.132311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:14.132311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:14.132311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:14.132311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:14.132811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:14.132811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:14.132811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:18.577522 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:18.579591 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:18.579591 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:18.579591 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:18.606293 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:18.606447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:18.608157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:18.608157 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:18.608157 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:18.608157 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:18.608157 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:18.608157 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:18.608157 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:18.608157 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:18.608157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:18.608157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:18.608157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:18.608157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:18.608157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:18.608157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:18.608157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:18.608805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:18.608805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:18.608805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:18.608805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:18.608805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:18.608805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:18.608805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:18.608805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:18.608805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:18.608805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:18.608805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:18.608805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:18.608805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:18.608805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:18.609316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:18.609316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:18.609316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:18.609316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:18.609316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:18.609316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:18.609316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:18.609316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:18.609316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:18.609316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:18.609316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:18.609316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:18.609316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:18.609316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:18.609825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:18.610581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:18.610581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:18.610581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:18.610581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:18.610581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:18.610581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:18.610581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:18.610581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:18.610581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:19.097109 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:19.099758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:19.099758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:19.100076 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:19.100312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.100766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:19.100766 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:19.100766 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:19.100766 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:19.100766 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:19.100766 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:19.100766 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.100766 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:19.101578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.101578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:19.101578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:19.101578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:19.101578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.101578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:19.101578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:19.101578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.101578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:19.101578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:19.101578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:19.101578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.101807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:19.101807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:19.101807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:19.101807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:19.101807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:19.101807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:19.101807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:19.101807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:19.101807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:19.101807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:19.101807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:19.101807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:19.101807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:19.102020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.102020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:19.102933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.102933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:19.102933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:19.102933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:19.102933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:19.102933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:19.102933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.102933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:19.102933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:19.102933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.102933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:19.102933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:19.102933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.102933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:19.102933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:19.102933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.103222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:19.103222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:19.103222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.103222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:19.103222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:19.103222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:19.103222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.103222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:19.103222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:19.103222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:19.103222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:19.103222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:19.103222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:19.103222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:19.103222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:19.103222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:19.103222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:19.103222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:19.103222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:19.103549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:19.103549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.103549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:19.112123 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:19.112383 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:19.115423 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:19.115612 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:21:19.119535 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:19.119777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.120206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:19.120206 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:19.120206 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:19.120206 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:19.120206 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:19.120206 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:19.120206 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.120206 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:19.121010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.121010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:19.121010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:19.121010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:19.121010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.121010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:19.121010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:19.121010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:19.121010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.121010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:19.121010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:19.121010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:19.121010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:19.121010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:19.121010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:19.121341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:19.121341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:19.121341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:19.121341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:19.121341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:19.121341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:19.121341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:19.121341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.121341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:19.122106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.122106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:19.122106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:19.122106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:19.122106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.122106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:19.122106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:19.122106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:21:19.122106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:19.122106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:19.122106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.122106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:19.122358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:19.122358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:19.122358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.122358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:19.122358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:19.122358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:19.122358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:19.122358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:19.122358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:19.122358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:19.122358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:19.122358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:19.122358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:19.122358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:19.122626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:19.122626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:19.122626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.122626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:19.126458 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:19.128963 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:19.129769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.129769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:19.129769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:19.129769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:19.129769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:19.129769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:19.129769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.129769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:19.129769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:19.129769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.129769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:19.129769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:19.129769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.129769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:19.129769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:19.129769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.130090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:19.130090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:19.130090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:21:19.130090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:19.130090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:19.130090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.130090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:19.130090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:19.130090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:19.130090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.130090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:19.130090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:19.130090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:19.130090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:19.130090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:19.130090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:19.130090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:19.130090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:19.130392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:19.130392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:19.130392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:19.130392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:19.130392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:19.130392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.130392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:19.130714 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:19.130900 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:19.131063 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:19.131375 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:19.131611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.132609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:19.132609 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:19.132609 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:19.132609 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:19.132609 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:19.132609 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:19.132609 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.132609 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:19.133308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.133308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:19.133308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:19.133308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:19.133308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.133308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:19.133308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:19.133308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:19.133308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.133308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:19.133308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:19.133308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:19.133308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:19.133308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:19.133308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:19.133570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:19.133570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:19.133570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:19.133570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:19.133570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:19.133570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:19.133570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:19.133570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.133570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:19.134370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.134370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:19.134370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:19.134370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:19.134370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.134370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:19.134370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:19.134370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:19.134370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:19.134370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.134370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:19.134607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:19.134607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:19.134607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.134607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:19.134607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:19.134607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:19.134607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:19.134607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:19.134607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:19.134607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:19.134607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:19.134607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:19.134607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:19.134607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:19.134863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:19.134863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:19.134863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.134863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:19.135645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.135645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:19.135645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:19.135645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:19.135645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:19.135645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:19.135645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.135645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:19.135645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:19.135645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.135645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:19.135645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:19.135645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.135645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:19.135645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:19.135645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.135645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:19.135967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:19.135967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:19.135967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:19.135967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.135967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:19.135967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:19.135967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:19.135967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.135967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:19.135967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:19.135967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:19.135967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:19.135967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:19.135967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:19.135967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:19.135967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:19.135967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:19.135967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:19.135967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:19.136243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:19.136243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:19.136243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.136243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:19.136489 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:19.136643 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:19.136846 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:19.137123 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:19.137380 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:19.137526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.137930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:19.137930 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:19.137930 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:19.137930 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:19.137930 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:19.137930 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:19.137930 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.137930 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:19.138800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.138800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:19.138800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:19.138800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:19.138800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.138800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:19.138800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:19.138800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.138800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:19.138800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:19.138800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:19.138800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.139053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:19.139053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:19.139053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:19.139053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:19.139053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:19.139053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:19.139053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:19.139053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:19.139053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:19.139053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:19.139053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:19.139053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:19.139053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:19.139269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.139269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:19.140000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.140000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:19.140000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:19.140000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:19.140000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:19.140000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:19.140000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.140000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:19.140000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:19.140000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.140000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:19.140000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:19.140000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.140000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:19.140000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:19.140000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:19.140323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:19.140323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:19.140323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.140323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:19.140323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:19.140323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:19.140323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:19.140323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:19.140323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:19.140323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:19.140323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:19.140323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:19.140323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:19.140323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:19.140323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:19.140323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:19.140323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:19.140323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:19.140323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:19.140650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:19.140650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:19.140650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:23.578390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:23.578959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:23.578959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:23.579549 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:23.602383 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:23.602548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:23.602996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:23.602996 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:23.602996 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:23.602996 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:23.602996 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:23.602996 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:23.602996 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:23.602996 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:23.603730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.024 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:23.603730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:23.603730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:23.603730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:23.603730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:23.603730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:23.603730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:23.603730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:23.603730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:23.603730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:23.603730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:23.603730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:23.604169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:23.604169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:23.604169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:23.604169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:23.604169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:23.604169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:23.604169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:23.604169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:23.604169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:23.604169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:23.604169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:23.604169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:23.604169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:23.604617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:23.604617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:23.604810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:23.604810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:23.604810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:23.604810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:23.604810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:23.604810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:23.604810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:23.604810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:23.604810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:23.604810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:23.604810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:23.604810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:23.604810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:23.604810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:23.604810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:23.604810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:23.605165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:23.605165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:23.605165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:23.605165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:23.605165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:23.605165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:23.605165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:23.605165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:23.605165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:23.605165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:23.605165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:23.605165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:23.605165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:23.605165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:23.605165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:23.605165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:23.605165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:23.605165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:23.605165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:23.605523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:23.605523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:23.605523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:24.099900 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:24.100371 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:24.100503 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:24.101818 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:24.101997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.102457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:24.102457 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:24.102457 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:24.102457 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:24.102457 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:24.102457 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:24.102457 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.102457 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:24.103350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.103350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:24.103350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:24.103350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:24.103350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.103350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:24.103350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:24.103350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.103350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:24.103350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:24.103350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:24.103350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.103722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:24.103722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:24.103722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:24.103722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:24.103722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:24.103722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:24.103722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:24.103722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:24.103722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:24.103722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:24.103722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:24.103722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:24.103722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:24.104057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.104057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:24.104120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.104120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:24.104120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:24.104120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:24.104120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:24.104120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:24.104120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.104120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:24.104120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:24.104120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.104120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:24.104120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:24.104120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.104120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:24.104120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:24.104120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.104568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:24.104568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:24.104568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.104568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:24.104568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:24.104568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:24.104568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.104568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:24.104568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:24.104568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:24.104568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:24.104568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:24.104568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:24.104568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:24.104568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:24.104568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:24.104568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:24.104568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:24.104568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:24.105058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:24.105058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.105058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:24.112702 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:24.113027 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:24.113164 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:24.113214 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:21:24.114176 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:24.114176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.114298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:24.114298 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:24.114298 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:24.114298 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:24.114298 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:24.114298 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:24.114298 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.114298 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:24.114894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.114894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:24.114894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:24.114894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:24.114894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.114894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:24.114894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:24.114894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:24.114894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.114894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:24.114894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:24.114894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:24.114894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:24.114894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:24.114894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:24.115319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:24.115319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:24.115319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:24.115319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:24.115319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:24.115319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:24.115319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:24.115319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.115319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:24.115713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.115713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:24.115713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:24.115713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:24.115713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.115713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:24.115713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:24.115713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:21:24.115713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:24.115713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:24.115713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.115713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:24.116032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:24.116032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:24.116032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.116032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:24.116032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:24.116032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:24.116032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:24.116032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:24.116032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:24.116032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:24.116032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:24.116032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:24.116032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:24.116032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:24.116367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:24.116367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:24.116367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.116367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:24.116725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.116725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:24.116725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:24.116725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:24.116725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:24.116725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:24.116725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.116725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:24.116725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:24.116725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.116725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:24.116725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:24.116725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.116725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:24.116725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:24.116725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.117240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:24.117240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:24.117240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:21:24.117240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:24.117240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:24.117240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.117240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:24.117240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:24.117240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:24.117240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.117240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:24.117240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:24.117240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:24.117240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:24.117240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:24.117240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:24.117240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:24.117240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:24.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:24.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:24.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:24.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:24.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:24.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:24.130831 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:24.131131 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:24.131243 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:24.131388 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:24.132150 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:24.132150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.132294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:24.132294 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:24.132294 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:24.132294 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:24.132294 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:24.132294 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:24.132294 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.132294 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:24.132916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.132916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:24.132916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:24.132916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:24.132916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.132916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:24.132916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:24.132916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:24.132916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.132916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:24.132916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:24.132916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:24.132916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:24.132916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:24.132916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:24.133147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:24.133147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:24.133147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:24.133147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:24.133147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:24.133147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:24.133147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:24.133147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.133147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:24.133724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.133724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:24.133724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:24.133724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:24.133724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.133724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:24.133724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:24.133724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:24.133724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:24.133724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.133724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:24.134066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:24.134066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:24.134066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.134066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:24.134066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:24.134066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:24.134066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:24.134066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:24.134066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:24.134066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:24.134066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:24.134066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:24.134066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:24.134066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:24.134429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:24.134429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:24.134429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.134429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:24.135293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.135293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:24.135293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:24.135293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:24.135293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:24.135293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:24.135293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.135293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:24.135293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:24.135293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.135293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:24.135293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:24.135293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.135293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:24.135293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:24.135293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.135293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:24.135719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:24.135719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:24.135719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:24.135719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.135719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:24.135719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:24.135719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:24.135719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.135719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:24.135719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:24.135719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:24.135719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:24.135719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:24.135719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:24.135719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:24.135719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:24.135719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:24.135719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:24.135719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:24.136063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:24.136063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:24.136063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.136063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:24.137307 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:24.137307 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:24.137307 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:24.137307 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:24.137307 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:24.137486 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:24.137613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.138054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:24.138054 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:24.138054 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:24.138054 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:24.138054 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:24.138054 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:24.138054 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.138054 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:24.138606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.138606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:24.138606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:24.138606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:24.138606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.138606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:24.138606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:24.138606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.138606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:24.138606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:24.138606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:24.138606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.138822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:24.138822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:24.138822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:24.138822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:24.138822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:24.138822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:24.138822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:24.138822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:24.138822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:24.138822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:24.138822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:24.138822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:24.138822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:24.139054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.139054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:24.139244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.139244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:24.139244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:24.139244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:24.139244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:24.139244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:24.139244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.139244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:24.139244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:24.139244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.139244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:24.139244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:24.139244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.139244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:24.139244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:24.139244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:24.139531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:24.139531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:24.139531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.139531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:24.139531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:24.139531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:24.139531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:24.139531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:24.139531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:24.139531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:24.139531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:24.139531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:24.139531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:24.139531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:24.139531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:24.139531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:24.139531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:24.139531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:24.139531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:24.139835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:24.139835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:24.139835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:28.578859 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:28.579328 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:28.579328 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:28.579815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:28.603072 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:28.603200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:28.603628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:28.603628 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:28.603628 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:28.603628 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:28.603628 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:28.603628 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:28.603628 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:28.603628 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:28.604272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.024 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:28.604272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:28.604272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:28.604272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:28.604272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:28.604272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:28.604272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:28.604272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:28.604272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:28.604272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:28.604272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:28.604272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:28.604490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:28.604490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:28.604490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:28.604490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:28.604490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:28.604490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:28.604490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:28.604490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:28.604490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:28.604490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:28.604490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:28.604490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:28.604490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:28.604751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:28.604751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:28.604996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:28.604996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:28.604996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:28.604996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:28.604996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:28.604996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:28.604996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:28.604996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:28.604996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:28.604996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:28.604996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:28.604996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:28.604996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:28.604996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:28.604996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:28.604996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:28.605312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:28.605312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:28.605312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:28.605312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:28.605312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:28.605312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:28.605312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:28.605312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:28.605312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:28.605312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:28.605312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:28.605312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:28.605312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:28.605312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:28.605312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:28.605312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:28.605312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:28.605312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:28.605312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:28.605631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:28.605631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:28.605631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:29.100431 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:29.100718 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:29.100829 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:29.101269 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:29.101446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.101849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:29.101849 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:29.101849 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:29.101849 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:29.101849 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:29.101849 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:29.101849 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.101849 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:29.102463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.102463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:29.102463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:29.102463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:29.102463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.102463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:29.102463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:29.102463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.102463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:29.102463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:29.102463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:29.102463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.102774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:29.102774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:29.102774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:29.102774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:29.102774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:29.102774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:29.102774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:29.102774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:29.102774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:29.102774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:29.102774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:29.102774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:29.102774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:29.102998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.102998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:29.103116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.103116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:29.103116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:29.103116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:29.103116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:29.103116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:29.103116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.103116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:29.103116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:29.103116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.103116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:29.103116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:29.103116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.103116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:29.103116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:29.103116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.103451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:29.103451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:29.103451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.103451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:29.103451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:29.103451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:29.103451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.103451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:29.103451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:29.103451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:29.103451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:29.103451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:29.103451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:29.103451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:29.103451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:29.103451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:29.103451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:29.103451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:29.103451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:29.103747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:29.103747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.103747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:29.113067 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:29.113478 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:29.113603 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:29.113741 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:21:29.114058 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:29.114220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.114610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:29.114610 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:29.114610 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:29.114610 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:29.114610 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:29.114610 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:29.114610 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.114610 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:29.115416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.115416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:29.115416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:29.115416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:29.115416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.115416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:29.115416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:29.115416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:29.115416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.115416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:29.115416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:29.115416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:29.115416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:29.115416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:29.115416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:29.115940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:29.115940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:29.115940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:29.115940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:29.115940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:29.115940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:29.115940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:29.115940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.115940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:29.116352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.116352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:29.116352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:29.116352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:29.116352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.116352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:29.116352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:29.116352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:21:29.116352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:29.116352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:29.116352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.116352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:29.116790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:29.116790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:29.116790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.116790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:29.116790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:29.116790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:29.116790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:29.116790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:29.116790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:29.116790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:29.116790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:29.116790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:29.116790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:29.116790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:29.117217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:29.117217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:29.117217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.117217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:29.118000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.118000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:29.118000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:29.118000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:29.118000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:29.118000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:29.118000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.118000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:29.118000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:29.118000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.118000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:29.118000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:29.118000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.118000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:29.118000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:29.118000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.118504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:29.118504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:29.118504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:21:29.118504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:29.118504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:29.118504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.118504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:29.118504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:29.118504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:29.118504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.118504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:29.118504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:29.118504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:29.118504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:29.118504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:29.118504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:29.118504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:29.118504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:29.119025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:29.119025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:29.119025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:29.119025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:29.119025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:29.119025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.119025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:29.131246 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:29.131457 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:29.131583 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:29.131720 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:29.132025 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:29.132216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.132625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:29.132625 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:29.132625 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:29.132625 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:29.132625 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:29.132625 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:29.132625 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.132625 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:29.133265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.133265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:29.133265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:29.133265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:29.133265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.133265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:29.133265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:29.133265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:29.133265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.133265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:29.133265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:29.133265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:29.133265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:29.133265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:29.133265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:29.133692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:29.133692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:29.133692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:29.133692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:29.133692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:29.133692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:29.133692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:29.133692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.133692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:29.134187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.134187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:29.134187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:29.134187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:29.134187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.134187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:29.134187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:29.134187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:29.134187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:29.134187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.134187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:29.134526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:29.134526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:29.134526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.134526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:29.134526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:29.134526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:29.134526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:29.134526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:29.134526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:29.134526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:29.134526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:29.134526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:29.134526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:29.134526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:29.134919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:29.134919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:29.134919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.134919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:29.135366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.135366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:29.135366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:29.135366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:29.135366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:29.135366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:29.135366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.135366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:29.135366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:29.135366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.135366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:29.135366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:29.135366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.135366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:29.135366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:29.135366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.135366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:29.135777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:29.135777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:29.135777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:29.135777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.135777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:29.135777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:29.135777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:29.135777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.135777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:29.135777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:29.135777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:29.135777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:29.135777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:29.135777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:29.135777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:29.135777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:29.135777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:29.135777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:29.135777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:29.136116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:29.136116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:29.136116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.136116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:29.136802 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:29.136992 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:29.137109 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:29.137263 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:29.137495 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:29.137687 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:29.137772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.138138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:29.138138 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:29.138138 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:29.138138 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:29.138138 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:29.138138 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:29.138138 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.138138 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:29.138869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.138869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:29.138869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:29.138869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:29.138869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.138869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:29.138869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:29.138869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.138869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:29.138869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:29.138869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:29.138869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.139206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:29.139206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:29.139206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:29.139206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:29.139206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:29.139206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:29.139206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:29.139206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:29.139206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:29.139206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:29.139206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:29.139206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:29.139206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:29.139563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.139563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:29.139844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.139844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:29.139844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:29.139844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:29.139844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:29.139844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:29.139844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.139844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:29.139844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:29.139844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.139844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:29.139844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:29.139844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.139844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:29.139844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:29.139844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:29.140131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:29.140131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:29.140131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.140131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:29.140131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:29.140131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:29.140131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:29.140131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:29.140131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:29.140131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:29.140131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:29.140131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:29.140131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:29.140131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:29.140131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:29.140131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:29.140131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:29.140131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:29.140131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:29.140521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:29.140521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:29.140521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:33.579275 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:33.579748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:33.579748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:33.580212 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:33.603444 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:33.603571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:33.604118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:33.604118 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:33.604118 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:33.604118 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:33.604118 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:33.604118 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:33.604118 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:33.604118 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:33.604858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:33.604858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:33.604858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:33.604858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:33.604858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:33.604858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:33.604858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:33.604858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:33.604858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:33.604858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:33.604858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:33.604858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:33.605159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:33.605159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:33.605159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:33.605159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:33.605159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:33.605159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:33.605159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:33.605159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:33.605159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:33.605159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:33.605159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:33.605159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:33.605159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:33.605433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:33.605433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:33.605736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:33.605736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:33.605736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:33.605736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:33.605736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:33.605736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:33.605736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:33.605736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:33.605736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:33.605736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:33.605736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:33.605736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:33.605736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:33.605736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:33.605736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:33.605736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:33.606092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:33.606092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:33.606092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:33.606092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:33.606092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:33.606092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:33.606092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:33.606092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:33.606092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:33.606092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:33.606092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:33.606092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:33.606092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:33.606092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:33.606092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:33.606092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:33.606092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:33.606092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:33.606092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:33.606741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:33.606741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:33.606741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:34.101526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:34.101526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:34.101526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:34.102168 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:34.102392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.102908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:34.102908 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:34.102908 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:34.102908 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:34.102908 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:34.102908 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:34.102908 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.102908 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:34.104185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.104185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:34.104185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:34.104185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:34.104185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.104185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:34.104185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:34.104185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.104185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:34.104185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:34.104185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:34.104185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.104486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:34.104486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:34.104486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:34.104486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:34.104486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:34.104486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:34.104486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:34.104486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:34.104486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:34.104486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:34.104486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:34.104486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:34.104486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:34.104717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.104717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:34.105115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.105115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:34.105115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:34.105115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:34.105115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:34.105115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:34.105115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.105115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:34.105115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:34.105115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.105115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:34.105115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:34.105115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.105115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:34.105115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:34.105115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.105421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:34.105421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:34.105421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.105421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:34.105421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:34.105421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:34.105421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.105421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:34.105421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:34.105421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:34.105421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:34.105421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:34.105421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:34.105421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:34.105421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:34.105421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:34.105421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:34.105421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:34.105421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:34.105731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:34.105731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.105731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:34.113709 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:34.114127 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:34.114285 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:34.114433 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:21:34.115413 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:34.115717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.116624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:34.116624 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:34.116624 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:34.116624 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:34.116624 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:34.116624 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:34.116624 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.116624 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:34.117569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.117569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:34.117569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:34.117569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:34.117569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.117569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:34.117569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:34.117569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:34.117569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.117569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:34.117569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:34.117569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:34.117569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:34.117569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:34.117569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:34.117980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:34.117980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:34.117980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:34.117980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:34.117980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:34.117980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:34.117980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:34.117980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.117980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:34.118610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.118610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:34.118610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:34.118610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:34.118610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.118610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:34.118610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:34.118610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:21:34.118610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:34.118610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:34.118610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.118610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:34.118926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:34.118926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:34.118926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.118926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:34.118926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:34.118926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:34.118926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:34.118926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:34.118926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:34.118926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:34.118926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:34.118926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:34.118926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:34.118926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:34.119223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:34.119223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:34.119223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.119223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:34.119628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.119628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:34.119628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:34.119628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:34.119628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:34.119628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:34.119628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.119628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:34.119628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:34.119628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.119628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:34.119628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:34.119628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.119628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:34.119628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:34.119628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.120062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:34.120062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:34.120062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:21:34.120062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:34.120062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:34.120062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.120062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:34.120062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:34.120062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:34.120062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.120062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:34.120062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:34.120062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:34.120062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:34.120062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:34.120062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:34.120062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:34.120062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:34.120557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:34.120557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:34.120557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:34.120557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:34.120557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:34.120557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.120557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:34.131540 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:34.131866 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:34.132005 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:34.132161 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:34.132785 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:34.133113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.133692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:34.133692 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:34.133692 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:34.133692 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:34.133692 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:34.133692 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:34.133692 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.133692 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:34.134647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.134647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:34.134647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:34.134647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:34.134647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.134647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:34.134647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:34.134647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:34.134647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.134647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:34.134647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:34.134647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:34.134647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:34.134647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:34.134647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:34.134930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:34.134930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:34.134930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:34.134930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:34.134930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:34.134930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:34.134930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:34.134930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.134930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:34.135620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.135620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:34.135620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:34.135620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:34.135620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.135620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:34.135620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:34.135620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:34.135620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:34.135620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.135620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:34.135848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:34.135848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:34.135848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.135848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:34.135848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:34.135848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:34.135848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:34.135848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:34.135848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:34.135848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:34.135848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:34.135848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:34.135848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:34.135848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:34.136098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:34.136098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:34.136098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.136098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:34.136468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.136468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:34.136468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:34.136468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:34.136468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:34.136468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:34.136468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.136468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:34.136468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:34.136468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.136468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:34.136468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:34.136468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.136468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:34.136468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:34.136468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.136468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:34.136784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:34.136784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:34.136784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:34.136784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.136784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:34.136784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:34.136784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:34.136784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.136784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:34.136784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:34.136784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:34.136784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:34.136784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:34.136784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:34.136784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:34.136784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:34.136784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:34.136784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:34.136784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:34.137102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:34.137102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:34.137102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.137102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:34.137174 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:34.137396 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:34.137473 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:34.137598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:34.137958 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:34.138203 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:34.138352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.138852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:34.138852 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:34.138852 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:34.138852 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:34.138852 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:34.138852 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:34.138852 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.138852 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:34.139415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.139415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:34.139415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:34.139415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:34.139415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.139415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:34.139415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:34.139415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.139415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:34.139415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:34.139415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:34.139415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.139640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:34.139640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:34.139640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:34.139640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:34.139640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:34.139640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:34.139640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:34.139640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:34.139640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:34.139640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:34.139640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:34.139640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:34.139640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:34.139847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.139847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:34.140032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.140032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:34.140032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:34.140032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:34.140032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:34.140032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:34.140032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.140032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:34.140032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:34.140032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.140032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:34.140032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:34.140032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.140032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:34.140032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:34.140032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:34.140313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:34.140313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:34.140313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.140313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:34.140313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:34.140313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:34.140313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:34.140313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:34.140313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:34.140313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:34.140313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:34.140313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:34.140313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:34.140313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:34.140313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:34.140313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:34.140313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:34.140313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:34.140313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:34.140646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:34.140646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:34.140646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:38.579695 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:38.580173 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:38.580173 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:38.580782 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:38.605294 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:38.605497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:38.606049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:38.606049 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:38.606049 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:38.606049 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:38.606049 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:38.606049 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:38.606049 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:38.606049 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:38.606791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:38.606791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:38.606791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:38.606791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:38.606791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:38.606791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:38.606791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:38.606791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:38.606791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:38.606791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:38.606791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:38.606791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:38.607037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:38.607037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:38.607037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:38.607037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:38.607037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:38.607037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:38.607037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:38.607037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:38.607037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:38.607037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:38.607037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:38.607037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:38.607037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:38.607226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:38.607226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:38.607502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:38.607502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:38.607502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:38.607502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:38.607502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:38.607502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:38.607502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:38.607502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:38.607502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:38.607502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:38.607502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:38.607502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:38.607502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:38.607502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:38.607502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:38.607502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:38.607765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:38.607765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:38.607765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:38.607765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:38.607765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:38.607765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:38.607765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:38.607765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:38.607765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:38.607765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:38.607765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:38.607765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:38.607765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:38.607765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:38.607765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:38.607765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:38.607765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:38.607765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:38.607765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:38.608119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:38.608119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:38.608119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:39.101314 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:39.101831 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:39.102063 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:39.102702 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:39.102891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.103602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:39.103602 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:39.103602 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:39.103602 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:39.103602 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:39.103602 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:39.103602 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.103602 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:39.104740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.104740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:39.104740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:39.104740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:39.104740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.104740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:39.104740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:39.104740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.104740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:39.104740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:39.104740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:39.104740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.104985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:39.104985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:39.104985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:39.104985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:39.104985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:39.104985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:39.104985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:39.104985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:39.104985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:39.104985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:39.104985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:39.104985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:39.104985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:39.105195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.105195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:39.105603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.105603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:39.105603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:39.105603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:39.105603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:39.105603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:39.105603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.105603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:39.105603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:39.105603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.105603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:39.105603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:39.105603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.105603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:39.105603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:39.105603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.105870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:39.105870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:39.105870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.105870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:39.105870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:39.105870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:39.105870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.105870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:39.105870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:39.105870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:39.105870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:39.105870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:39.105870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:39.105870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:39.105870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:39.105870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:39.105870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:39.105870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:39.105870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:39.106176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:39.106176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.106176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:39.114073 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:39.114348 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:39.114510 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:39.114679 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:21:39.115241 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:39.115439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.115931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:39.115931 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:39.115931 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:39.115931 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:39.115931 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:39.115931 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:39.115931 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.115931 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:39.116668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.116668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:39.116668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:39.116668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:39.116668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.116668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:39.116668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:39.116668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:39.116668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.116668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:39.116668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:39.116668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:39.116668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:39.116668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:39.116668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:39.116995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:39.116995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:39.116995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:39.116995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:39.116995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:39.116995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:39.116995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:39.116995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.116995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:39.117236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.117236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:39.117236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:39.117236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:39.117236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.117236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:39.117236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:39.117236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:21:39.117236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:39.117236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:39.117236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.117236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:39.117471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:39.117471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:39.117471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.117471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:39.117471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:39.117471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:39.117471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:39.117471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:39.117471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:39.117471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:39.117471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:39.117471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:39.117471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:39.117471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:39.117752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:39.117752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:39.117752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.117752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:39.117928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.117928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:39.117928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:39.117928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:39.117928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:39.117928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:39.117928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.117928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:39.117928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:39.117928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.117928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:39.117928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:39.117928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.117928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:39.117928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:39.117928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.118356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:39.118356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:39.118356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:21:39.118356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:39.118356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:39.118356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.118356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:39.118356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:39.118356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:39.118356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.118356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:39.118356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:39.118356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:39.118356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:39.118356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:39.118356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:39.118356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:39.118356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:39.118790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:39.118790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:39.118790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:39.118790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:39.118790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:39.118790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.118790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:39.131944 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:39.132122 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:39.132245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:39.132390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:39.132917 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:39.133184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.133703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:39.133703 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:39.133703 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:39.133703 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:39.133703 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:39.133703 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:39.133703 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.133703 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:39.134353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.134353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:39.134353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:39.134353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:39.134353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.134353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:39.134353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:39.134353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:39.134353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.134353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:39.134353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:39.134353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:39.134353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:39.134353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:39.134353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:39.134640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:39.134640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:39.134640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:39.134640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:39.134640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:39.134640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:39.134640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:39.134640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.134640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:39.134987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.134987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:39.134987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:39.134987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:39.134987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.134987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:39.134987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:39.134987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:39.134987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:39.134987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.134987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:39.135178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:39.135178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:39.135178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.135178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:39.135178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:39.135178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:39.135178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:39.135178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:39.135178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:39.135178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:39.135178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:39.135178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:39.135178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:39.135178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:39.135405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:39.135405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:39.135405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.135405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:39.135688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.135688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:39.135688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:39.135688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:39.135688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:39.135688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:39.135688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.135688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:39.135688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:39.135688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.135688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:39.135688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:39.135688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.135688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:39.135688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:39.135688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.135688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:39.136016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:39.136016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:39.136016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:39.136016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.136016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:39.136016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:39.136016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:39.136016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.136016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:39.136016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:39.136016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:39.136016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:39.136016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:39.136016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:39.136016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:39.136016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:39.136016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:39.136016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:39.136016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:39.136313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:39.136313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:39.136313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.136313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:39.137445 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:39.137616 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:39.137700 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:39.137840 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:39.138321 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:39.138586 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:39.138711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.139221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:39.139221 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:39.139221 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:39.139221 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:39.139221 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:39.139221 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:39.139221 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.139221 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:39.139795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.139795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:39.139795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:39.139795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:39.139795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.139795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:39.139795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:39.139795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.139795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:39.139795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:39.139795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:39.139795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.140033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:39.140033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:39.140033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:39.140033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:39.140033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:39.140033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:39.140033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:39.140033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:39.140033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:39.140033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:39.140033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:39.140033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:39.140033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:39.140232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.140232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:39.140499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.140499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:39.140499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:39.140499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:39.140499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:39.140499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:39.140499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.140499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:39.140499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:39.140499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.140499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:39.140499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:39.140499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.140499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:39.140499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:39.140499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:39.140769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:39.140769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:39.140769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.140769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:39.140769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:39.140769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:39.140769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:39.140769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:39.140769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:39.140769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:39.140769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:39.140769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:39.140769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:39.140769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:39.140769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:39.140769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:39.140769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:39.140769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:39.140769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:39.141093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:39.141093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:39.141093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:43.580076 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:43.580672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:43.580672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:43.581025 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:43.603984 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:43.604189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:43.604716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:43.604716 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:43.604716 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:43.604716 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:43.604716 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:43.604716 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:43.604716 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:43.604716 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:43.605548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.024 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:43.605548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:43.605548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:43.605548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:43.605548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:43.605548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:43.605548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:43.605548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:43.605548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:43.605548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:43.605548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:43.605548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:43.605779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:43.605779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:43.605779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:43.605779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:43.605779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:43.605779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:43.605779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:43.605779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:43.605779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:43.605779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:43.605779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:43.605779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:43.605779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:43.606001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:43.606001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:43.606246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:43.606246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:43.606246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:43.606246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:43.606246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:43.606246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:43.606246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:43.606246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:43.606246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:43.606246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:43.606246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:43.606246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:43.606246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:43.606246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:43.606246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:43.606246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:43.607029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:43.607029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:43.607029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:43.607029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:43.607029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:43.607029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:43.607029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:43.607029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:43.607029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:43.607029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:43.607029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:43.607029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:43.607029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:43.607029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:43.607029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:43.607029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:43.607029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:43.607029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:43.607029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:43.607638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:43.607638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:43.607638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:44.101805 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:44.102191 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:44.102333 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:44.102838 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:44.103033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.103517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:44.103517 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:44.103517 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:44.103517 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:44.103517 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:44.103517 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:44.103517 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.103517 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:44.104207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.104207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:44.104207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:44.104207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:44.104207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.104207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:44.104207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:44.104207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.104207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:44.104207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:44.104207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:44.104207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.104449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:44.104449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:44.104449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:44.104449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:44.104449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:44.104449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:44.104449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:44.104449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:44.104449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:44.104449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:44.104449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:44.104449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:44.104449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:44.104645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.104645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:44.104936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.104936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:44.104936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:44.104936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:44.104936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:44.104936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:44.104936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.104936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:44.104936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:44.104936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.104936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:44.104936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:44.104936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.104936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:44.104936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:44.104936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.105197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:44.105197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:44.105197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.105197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:44.105197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:44.105197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:44.105197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.105197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:44.105197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:44.105197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:44.105197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:44.105197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:44.105197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:44.105197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:44.105197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:44.105197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:44.105197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:44.105197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:44.105197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:44.105554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:44.105554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.105554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:44.114402 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:44.114728 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:44.114892 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:44.115034 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:21:44.115365 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:44.115519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.115957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:44.115957 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:44.115957 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:44.115957 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:44.115957 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:44.115957 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:44.115957 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.115957 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:44.116801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.116801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:44.116801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:44.116801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:44.116801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.116801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:44.116801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:44.116801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:44.116801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.116801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:44.116801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:44.116801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:44.116801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:44.116801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:44.116801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:44.117101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:44.117101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:44.117101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:44.117101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:44.117101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:44.117101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:44.117101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:44.117101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.117101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:44.117685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.117685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:44.117685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:44.117685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:44.117685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.117685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:44.117685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:44.117685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:21:44.117685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:44.117685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:44.117685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.117685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:44.117917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:44.117917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:44.117917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.117917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:44.117917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:44.117917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:44.117917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:44.117917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:44.117917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:44.117917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:44.117917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:44.117917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:44.117917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:44.117917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:44.118128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:44.118128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:44.118128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.118128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:44.118749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.118749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:44.118749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:44.118749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:44.118749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:44.118749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:44.118749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.118749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:44.118749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:44.118749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.118749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:44.118749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:44.118749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.118749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:44.118749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:44.118749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.119023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:44.119023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:44.119023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:21:44.119023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:44.119023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:44.119023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.119023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:44.119023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:44.119023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:44.119023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.119023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:44.119023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:44.119023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:44.119023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:44.119023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:44.119023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:44.119023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:44.119023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:44.119313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:44.119313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:44.119313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:44.119313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:44.119313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:44.119313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.119313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:44.132192 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:44.132480 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:44.132594 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:44.132740 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:44.133085 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:44.133285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.133709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:44.133709 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:44.133709 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:44.133709 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:44.133709 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:44.133709 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:44.133709 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.133709 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:44.134515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.134515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:44.134515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:44.134515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:44.134515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.134515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:44.134515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:44.134515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:44.134515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.134515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:44.134515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:44.134515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:44.134515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:44.134515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:44.134515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:44.134757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:44.134757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:44.134757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:44.134757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:44.134757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:44.134757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:44.134757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:44.134757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.134757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:44.135441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.135441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:44.135441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:44.135441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:44.135441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.135441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:44.135441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:44.135441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:44.135441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:44.135441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.135441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:44.135647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:44.135647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:44.135647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.135647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:44.135647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:44.135647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:44.135647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:44.135647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:44.135647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:44.135647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:44.135647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:44.135647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:44.135647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:44.135647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:44.135852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:44.135852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:44.135852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.135852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:44.136533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.136533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:44.136533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:44.136533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:44.136533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:44.136533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:44.136533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.136533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:44.136533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:44.136533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.136533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:44.136533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:44.136533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.136533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:44.136533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:44.136533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.136533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:44.136840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:44.136840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:44.136840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:44.136840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.136840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:44.136840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:44.136840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:44.136840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.136840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:44.136840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:44.136840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:44.136840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:44.136840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:44.136840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:44.136840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:44.136840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:44.136840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:44.136840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:44.136840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:44.137147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:44.137147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:44.137147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.137147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:44.137679 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:44.137907 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:44.137996 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:44.138110 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:44.138377 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:44.138559 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:44.138641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.139068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:44.139068 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:44.139068 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:44.139068 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:44.139068 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:44.139068 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:44.139068 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.139068 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:44.139931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.139931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:44.139931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:44.139931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:44.139931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.139931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:44.139931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:44.139931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.139931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:44.139931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:44.139931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:44.139931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.140169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:44.140169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:44.140169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:44.140169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:44.140169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:44.140169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:44.140169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:44.140169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:44.140169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:44.140169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:44.140169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:44.140169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:44.140169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:44.140391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.140391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:44.140918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.140918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:44.140918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:44.140918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:44.140918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:44.140918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:44.140918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.140918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:44.140918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:44.140918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.140918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:44.140918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:44.140918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.140918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:44.140918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:44.140918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:44.141169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:44.141169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:44.141169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.141169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:44.141169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:44.141169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:44.141169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:44.141169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:44.141169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:44.141169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:44.141169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:44.141169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:44.141169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:44.141169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:44.141169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:44.141169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:44.141169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:44.141169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:44.141169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:44.141517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:44.141517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:44.141517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:48.580457 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:48.580846 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:48.580846 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:48.581691 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:48.582725 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=596,neutron:INSERT=1,neutron:UPDATE=2,neutron:DELETE=1 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:21:48.605423 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:48.605645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:48.606162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:48.606162 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:48.606162 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:48.606162 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:48.606162 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:48.606162 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:48.606162 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:48.606162 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:48.606946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:48.606946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:48.606946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:48.606946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:48.606946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:48.606946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:48.606946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:48.606946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:48.606946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:48.606946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:48.606946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:48.606946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:48.607293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:48.607293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:48.607293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:48.607293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:48.607293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:48.607293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:48.607293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:48.607293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:48.607293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:48.607293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:48.607293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:48.607293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:48.607293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:48.607632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:48.607632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:48.607855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:48.607855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:48.607855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:48.607855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:48.607855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:48.607855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:48.607855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:48.607855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:48.607855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:48.607855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:48.607855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:48.607855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:48.607855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:48.607855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:48.607855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:48.607855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:48.608163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:48.608163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:48.608163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:48.608163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:48.608163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:48.608163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:48.608163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:48.608163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:48.608163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:48.608163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:48.608163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:48.608163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:48.608163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:48.608163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:48.608163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:48.608163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:48.608163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:48.608163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:48.608163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:48.608478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:48.608478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:48.608478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:49.102758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:49.102758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:49.102758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:49.103156 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:49.103369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.104005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:49.104005 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:49.104005 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:49.104005 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:49.104005 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:49.104005 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:49.104005 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.104005 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:49.104829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.104829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:49.104829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:49.104829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:49.104829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.104829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:49.104829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:49.104829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.104829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:49.104829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:49.104829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:49.104829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.105304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:49.105304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:49.105304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:49.105304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:49.105304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:49.105304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:49.105304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:49.105304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:49.105304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:49.105304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:49.105304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:49.105304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:49.105304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:49.105681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.105681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:49.105896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.105896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:49.105896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:49.105896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:49.105896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:49.105896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:49.105896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.105896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:49.105896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:49.105896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.105896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:49.105896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:49.105896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.105896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:49.105896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:49.105896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.106370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:49.106370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:49.106370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.106370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:49.106370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:49.106370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:49.106370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.106370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:49.106370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:49.106370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:49.106370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:49.106370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:49.106370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:49.106370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:49.106370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:49.106370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:49.106370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:49.106370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:49.106370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:49.106849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:49.106849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.106849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:49.114730 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:49.114959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:49.115056 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:49.115172 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:21:49.115573 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:49.115769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.116217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:49.116217 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:49.116217 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:49.116217 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:49.116217 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:49.116217 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:49.116217 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.116217 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:49.116802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.116802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:49.116802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:49.116802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:49.116802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.116802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:49.116802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:49.116802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:49.116802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.116802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:49.116802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:49.116802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:49.116802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:49.116802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:49.116802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:49.117091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:49.117091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:49.117091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:49.117091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:49.117091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:49.117091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:49.117091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:49.117091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.117091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:49.117449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.117449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:49.117449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:49.117449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:49.117449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.117449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:49.117449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:49.117449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:21:49.117449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:49.117449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:49.117449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.117449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:49.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:49.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:49.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:49.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:49.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:49.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:49.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:49.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:49.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:49.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:49.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:49.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:49.117758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:49.118151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:49.118151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:49.118151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.118151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:49.118151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.118151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:49.118151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:49.118151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:49.118151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:49.118151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:49.118151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.118151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:49.118151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:49.118151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.118564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:49.118564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:49.118564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.118564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:49.118564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:49.118564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.118564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:49.118564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:49.118564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:21:49.118564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:49.118564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:49.118564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.118564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:49.118564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:49.118564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:49.118564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.118564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:49.118564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:49.118564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:49.119107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:49.119107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:49.119107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:49.119107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:49.119107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:49.119107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:49.119107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:49.119107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:49.119107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:49.119107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:49.119107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.119107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:49.132547 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:49.132806 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:49.132934 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:49.133049 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:49.133684 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:49.133684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.134016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:49.134016 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:49.134016 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:49.134016 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:49.134016 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:49.134016 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:49.134016 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.134016 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:49.134717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.134717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:49.134717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:49.134717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:49.134717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.134717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:49.134717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:49.134717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:49.134717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.134717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:49.134717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:49.134717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:49.134717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:49.134717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:49.134717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:49.134988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:49.134988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:49.134988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:49.134988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:49.134988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:49.134988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:49.134988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:49.134988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.134988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:49.135498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.135498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:49.135498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:49.135498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:49.135498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.135498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:49.135498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:49.135498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:49.135498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:49.135498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.135498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:49.135698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:49.135698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:49.135698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.135698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:49.135698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:49.135698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:49.135698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:49.135698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:49.135698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:49.135698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:49.135698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:49.135698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:49.135698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:49.135698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:49.135929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:49.135929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:49.135929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.135929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:49.136245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.136245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:49.136245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:49.136245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:49.136245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:49.136245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:49.136245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.136245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:49.136245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:49.136245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.136245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:49.136245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:49.136245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.136245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:49.136245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:49.136245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.136245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:49.136541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:49.136541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:49.136541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:49.136541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.136541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:49.136541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:49.136541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:49.136541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.136541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:49.136541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:49.136541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:49.136541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:49.136541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:49.136541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:49.136541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:49.136541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:49.136541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:49.136541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:49.136541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:49.136826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:49.136826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:49.136826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.136826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:49.137927 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:49.138103 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:49.138185 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:49.138332 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:49.138598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:49.138782 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:49.138891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.139276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:49.139276 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:49.139276 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:49.139276 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:49.139276 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:49.139276 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:49.139276 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.139276 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:49.140236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.140236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:49.140236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:49.140236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:49.140236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.140236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:49.140236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:49.140236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.140236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:49.140236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:49.140236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:49.140236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.140505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:49.140505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:49.140505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:49.140505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:49.140505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:49.140505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:49.140505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:49.140505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:49.140505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:49.140505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:49.140505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:49.140505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:49.140505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:49.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:49.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:49.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:49.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:49.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:49.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:49.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:49.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:49.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:49.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:49.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.140951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:49.140951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:49.140951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:49.140951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:49.140951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:49.140951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.140951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:49.140951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:49.140951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:49.140951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:49.140951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:49.140951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:49.140951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:49.140951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:49.140951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:49.140951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:49.140951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:49.140951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:49.140951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:49.141228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:49.141228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:49.141228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:49.141228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:49.141228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:49.141228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:53.580852 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:53.581313 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:53.581420 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:53.582020 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:53.609196 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:53.609447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:53.609985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:53.609985 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:53.609985 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:53.609985 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:53.609985 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:53.609985 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:53.609985 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:53.609985 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:53.610673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:53.610673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:53.610673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:53.610673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:53.610673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:53.610673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:53.610673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:53.610673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:53.610673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:53.610673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:53.610673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:53.610673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:53.610957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:53.610957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:53.610957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:53.610957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:53.610957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:53.610957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:53.610957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:53.610957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:53.610957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:53.610957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:53.610957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:53.610957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:53.610957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:53.611176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:53.611176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:53.611372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:53.611372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:53.611372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:53.611372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:53.611372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:53.611372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:53.611372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:53.611372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:53.611372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:53.611372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:53.611372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:53.611372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:53.611372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:53.611372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:53.611372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:53.611372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:53.611689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:53.611689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:53.611689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:53.611689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:53.611689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:53.611689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:53.611689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:53.611689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:53.611689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:53.611689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:53.611689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:53.611689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:53.611689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:53.611689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:53.611689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:53.611689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:53.611689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:53.611689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:53.611689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:53.612039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:53.612039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:53.612039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:54.102715 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:54.103057 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:54.103157 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:54.103708 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:54.103906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.104410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:54.104410 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:54.104410 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:54.104410 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:54.104410 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:54.104410 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:54.104410 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.104410 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:54.105146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.105146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:54.105146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:54.105146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:54.105146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.105146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:54.105146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:54.105146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.105146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:54.105146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:54.105146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:54.105146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.105454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:54.105454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:54.105454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:54.105454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:54.105454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:54.105454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:54.105454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:54.105454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:54.105454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:54.105454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:54.105454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:54.105454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:54.105454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:54.105689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.105689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:54.105950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.105950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:54.105950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:54.105950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:54.105950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:54.105950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:54.105950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.105950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:54.105950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:54.105950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.105950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:54.105950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:54.105950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.105950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:54.105950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:54.105950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.106311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:54.106311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:54.106311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.106311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:54.106311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:54.106311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:54.106311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.106311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:54.106311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:54.106311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:54.106311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:54.106311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:54.106311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:54.106311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:54.106311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:54.106311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:54.106311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:54.106311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:54.106311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:54.106669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:54.106669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.106669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:54.115434 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:54.116353 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:54.116353 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:54.116353 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:21:54.116353 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:54.116353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.116781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:54.116781 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:54.116781 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:54.116781 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:54.116781 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:54.116781 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:54.116781 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.116781 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:54.117522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.117522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:54.117522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:54.117522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:54.117522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.117522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:54.117522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:54.117522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:54.117522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.117522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:54.117522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:54.117522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:54.117522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:54.117522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:54.117522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:54.117886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:54.117886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:54.117886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:54.117886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:54.117886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:54.117886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:54.117886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:54.117886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.117886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:54.118179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.118179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:54.118179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:54.118179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:54.118179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.118179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:54.118179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:54.118179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:21:54.118179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:54.118179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:54.118179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.118179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:54.118492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:54.118492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:54.118492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.118492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:54.118492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:54.118492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:54.118492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:54.118492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:54.118492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:54.118492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:54.118492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:54.118492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:54.118492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:54.118492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:54.118760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:54.118760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:54.118760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.118760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:54.118850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.118850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:54.118850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:54.118850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:54.118850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:54.118850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:54.118850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.118850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:54.118850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:54.118850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.118850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:54.118850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:54.118850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.118850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:54.118850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:54.118850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.119158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:54.119158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:54.119158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:21:54.119158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:54.119158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:54.119158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.119158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:54.119158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:54.119158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:54.119158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.119158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:54.119158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:54.119158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:54.119158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:54.119158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:54.119158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:54.119158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:54.119158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:54.119539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:54.119539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:54.119539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:54.119539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:54.119539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:54.119539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.119539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:54.133549 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:54.133762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:54.133917 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:54.133999 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:54.134340 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:54.134541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.134993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:54.134993 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:54.134993 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:54.134993 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:54.134993 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:54.134993 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:54.134993 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.134993 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:54.135567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.135567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:54.135567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:54.135567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:54.135567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.135567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:54.135567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:54.135567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:54.135567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.135567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:54.135567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:54.135567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:54.135567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:54.135567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:54.135567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:54.135902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:54.135902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:54.135902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:54.135902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:54.135902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:54.135902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:54.135902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:54.135902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.135902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:54.136156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.136156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:54.136156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:54.136156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:54.136156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.136156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:54.136156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:54.136156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:54.136156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:54.136156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.136156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:54.136701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:54.136701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:54.136701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.136701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:54.136701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:54.136701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:54.136701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:54.136701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:54.136701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:54.136701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:54.136701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:54.136701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:54.136701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:54.136701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:54.137401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:54.137401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:54.137401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.137401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:54.137401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.137401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:54.137401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:54.137401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:54.137401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:54.137401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:54.137401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.137401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:54.137401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:54.137401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.137726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:54.137726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:54.137726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.137726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:54.137726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:54.137726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.137726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:54.137726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:54.137726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:54.137726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:54.137726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.137726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:54.137726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:54.137726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:54.137726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.137726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:54.137726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:54.137726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:54.137726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:54.138498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:54.138498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:54.138498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:54.138498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:54.138498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:54.138498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:54.138498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:54.138498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:54.138498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:54.138498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.138498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:54.138498 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:54.138498 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:54.139087 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:54.139087 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:54.139087 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:54.139087 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:54.139361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.139558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:54.139558 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:54.139558 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:54.139558 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:54.139558 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:54.139558 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:54.139558 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.139558 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:54.140098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.140098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:54.140098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:54.140098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:54.140098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.140098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:54.140098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:54.140098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.140098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:54.140098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:54.140098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:54.140098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.140340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:54.140340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:54.140340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:54.140340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:54.140340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:54.140340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:54.140340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:54.140340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:54.140340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:54.140340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:54.140340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:54.140340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:54.140340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:54.140535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.140535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:54.140758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.140758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:54.140758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:54.140758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:54.140758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:54.140758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:54.140758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.140758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:54.140758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:54.140758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.140758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:54.140758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:54.140758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.140758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:54.140758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:54.140758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:54.141069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:54.141069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:54.141069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.141069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:54.141069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:54.141069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:54.141069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:54.141069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:54.141069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:54.141069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:54.141069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:54.141069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:54.141069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:54.141069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:54.141069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:54.141069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:54.141069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:54.141069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:54.141069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:54.141400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:54.141400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:54.141400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:58.581748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:58.582147 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:58.582147 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:58.582493 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:58.604751 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:58.604917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:58.605326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:58.605326 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:58.605326 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:58.605326 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:58.605326 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:58.605326 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:58.605326 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:58.605326 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:58.605964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.023 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:58.605964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:58.605964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:58.605964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:58.605964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:58.605964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:58.605964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:58.605964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:58.605964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:58.605964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:58.605964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:58.605964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:58.606291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:58.606291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:58.606291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:58.606291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:58.606291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:58.606291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:58.606291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:58.606291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:58.606291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:58.606291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:58.606291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:58.606291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:58.606291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:58.606584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:58.606584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:58.606642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:58.606642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:58.606642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:58.606642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:58.606642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:58.606642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:58.606642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:58.606642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:58.606642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:58.606642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:58.606642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:58.606642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:58.606642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:58.606642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:58.606642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:58.606642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:58.607038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:21:58.607038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:58.607038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:58.607038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:58.607038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:58.607038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:58.607038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:58.607038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:58.607038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:58.607038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:58.607038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:58.607038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:58.607038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:58.607038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:58.607038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:58.607038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:58.607038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:58.607038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:58.607038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:58.607485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:58.607485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:58.607485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:59.103183 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:59.103338 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:59.103448 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:59.103893 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:59.104051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.104506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:59.104506 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:59.104506 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:59.104506 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:59.104506 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:59.104506 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:59.104506 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.104506 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:59.105116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.105116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:59.105116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:59.105116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:59.105116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.105116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:59.105116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:59.105116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.105116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:59.105116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:59.105116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:59.105116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.105455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:59.105455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:59.105455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:59.105455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:59.105455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:59.105455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:59.105455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:59.105455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:59.105455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:59.105455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:59.105455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:59.105455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:59.105455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:59.105750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.105750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:59.105808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.105808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:59.105808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:59.105808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:59.105808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:59.105808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:59.105808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.105808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:59.105808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:59.105808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.105808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:59.105808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:59.105808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.105808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:59.105808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:59.105808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.106198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:21:59.106198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:59.106198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.106198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:59.106198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:59.106198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:59.106198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.106198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:59.106198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:59.106198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:59.106198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:59.106198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:59.106198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:59.106198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:59.106198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:59.106198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:59.106198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:59.106198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:59.106198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:59.106646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:59.106646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.106646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:59.115672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:59.115840 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:59.115924 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:59.116005 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:21:59.116261 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:59.116431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:59.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:59.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:59.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:59.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:59.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:59.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.116795 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:59.117520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.117520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:59.117520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:59.117520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:59.117520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.117520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:59.117520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:59.117520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:59.117520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.117520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:59.117520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:59.117520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:59.117520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:59.117520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:59.117520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:59.117899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:59.117899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:59.117899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:59.117899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:59.117899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:59.117899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:59.117899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:59.117899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.117899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:59.118346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.118346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:59.118346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:59.118346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:59.118346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.118346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:59.118346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:59.118346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:21:59.118346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:59.118346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:59.118346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.118346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:59.118657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:59.118657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:59.118657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.118657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:59.118657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:59.118657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:59.118657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:59.118657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:59.118657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:59.118657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:59.118657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:59.118657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:59.118657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:59.118657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:59.118981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:59.118981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:59.118981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.118981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:59.119333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.119333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:59.119333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:59.119333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:59.119333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:59.119333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:59.119333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.119333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:59.119333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:59.119333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.119333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:59.119333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:59.119333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.119333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:59.119333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:59.119333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.120098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:21:59.120098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:21:59.120098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:21:59.120098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:59.120098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:59.120098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.120098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:59.120098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:59.120098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:59.120098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.120098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:59.120098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:59.120098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:59.120098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:59.120098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:59.120098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:59.120098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:59.120098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:59.120903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:59.120903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:59.120903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:59.120903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:59.120903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:59.120903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.120903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:59.133756 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:59.133903 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:59.133979 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:59.134067 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:59.134331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:59.134516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.134846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:59.134846 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:59.134846 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:59.134846 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:59.134846 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:59.134846 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:59.134846 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.134846 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:59.135319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.135319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:21:59.135319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:59.135319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:59.135319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.135319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:59.135319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:59.135319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:21:59.135319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.135319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:21:59.135319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:59.135319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:21:59.135319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:59.135319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:21:59.135319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:21:59.135750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:59.135750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:21:59.135750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:59.135750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:21:59.135750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:21:59.135750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:59.135750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:21:59.135750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.135750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:21:59.135965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.135965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:59.135965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:59.135965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:59.135965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.135965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:59.135965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:59.135965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:59.135965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:59.135965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.135965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:59.136242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:59.136242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:59.136242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.136242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:59.136242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:59.136242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:59.136242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:59.136242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:59.136242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:59.136242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:59.136242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:59.136242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:59.136242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:59.136242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:59.136572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:59.136572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:59.136572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.136572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:59.136572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.136572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:59.136572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:59.136572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:59.136572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:59.136572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:59.136572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.136572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:59.136572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:59.136572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.136906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:59.136906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:59.136906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.136906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:59.136906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:59.136906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.136906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:21:59.136906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:21:59.136906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:21:59.136906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:21:59.136906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.136906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:59.136906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:59.136906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:59.136906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.136906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:59.136906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:59.136906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:59.136906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:59.137355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:59.137355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:59.137355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:59.137355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:59.137355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:59.137355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:59.137355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:59.137355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:59.137355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:59.137355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.137355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:21:59.138439 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:21:59.138563 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:21:59.138636 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:21:59.138739 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:21:59.138968 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:59.139145 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:21:59.139224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.139590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:21:59.139590 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:59.139590 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:21:59.139590 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:21:59.139590 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:59.139590 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:21:59.139590 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.139590 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:21:59.140129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.140129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:21:59.140129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:59.140129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:21:59.140129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.140129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:59.140129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:59.140129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.140129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:59.140129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:59.140129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:21:59.140129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.140434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:21:59.140434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:59.140434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:21:59.140434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:59.140434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:21:59.140434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:21:59.140434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:59.140434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:21:59.140434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:59.140434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:21:59.140434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:21:59.140434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:59.140434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:21:59.140736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.140736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:21:59.140736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.140736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:21:59.140736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:21:59.140736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:21:59.140736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:21:59.140736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:21:59.140736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.140736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:21:59.140736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:21:59.140736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.140736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:21:59.140736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:21:59.140736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:21:59.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:21:59.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:21:59.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:21:59.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:21:59.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:59.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:21:59.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:21:59.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:21:59.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:21:59.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:21:59.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:21:59.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:21:59.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:21:59.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:21:59.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:21:59.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:21:59.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:21:59.141532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:21:59.141532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:21:59.141532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:21:59.141532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:21:59.141532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:21:59.141532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:03.581842 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:03.582382 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:03.582382 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:03.582721 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:03.621181 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:03.621386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:03.621767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:03.621767 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:03.621767 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:03.621767 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:03.621767 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:03.621767 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:03.621767 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:03.621767 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:03.622394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:03.622394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:03.622394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:03.622394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:03.622394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:03.622394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:03.622394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:03.622394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:03.622394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:03.622394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:03.622394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:03.622394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:03.622617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:03.622617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:03.622617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:03.622617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:03.622617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:03.622617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:03.622617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:03.622617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:03.622617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:03.622617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:03.622617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:03.622617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:03.622617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:03.622815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:03.622815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:03.623059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:03.623059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:03.623059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:03.623059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:03.623059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:03.623059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:03.623059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:03.623059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:03.623059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:03.623059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:03.623059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:03.623059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:03.623059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:03.623059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:03.623059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:03.623059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:03.623338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:03.623338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:03.623338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:03.623338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:03.623338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:03.623338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:03.623338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:03.623338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:03.623338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:03.623338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:03.623338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:03.623338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:03.623338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:03.623338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:03.623338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:03.623338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:03.623338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:03.623338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:03.623338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:03.623617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:03.623617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:03.623617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:04.103588 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:04.103866 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:04.103866 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:04.104535 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:04.104765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.105274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:04.105274 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:04.105274 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:04.105274 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:04.105274 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:04.105274 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:04.105274 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.105274 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:04.105942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.105942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:04.105942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:04.105942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:04.105942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.105942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:04.105942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:04.105942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.105942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:04.105942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:04.105942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:04.105942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.106188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:04.106188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:04.106188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:04.106188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:04.106188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:04.106188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:04.106188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:04.106188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:04.106188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:04.106188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:04.106188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:04.106188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:04.106188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:04.106445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.106445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:04.106727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.106727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:04.106727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:04.106727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:04.106727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:04.106727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:04.106727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.106727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:04.106727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:04.106727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.106727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:04.106727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:04.106727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.106727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:04.106727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:04.106727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.107401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:04.107401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:04.107401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.107401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:04.107401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:04.107401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:04.107401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.107401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:04.107401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:04.107401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:04.107401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:04.107401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:04.107401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:04.107401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:04.107401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:04.107401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:04.107401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:04.107401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:04.107401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:04.107972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:04.107972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.107972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:04.116097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:04.116316 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:04.116388 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:04.116472 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:22:04.116937 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:04.117184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.117668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:04.117668 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:04.117668 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:04.117668 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:04.117668 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:04.117668 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:04.117668 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.117668 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:04.118270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.118270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:04.118270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:04.118270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:04.118270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.118270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:04.118270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:04.118270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:04.118270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.118270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:04.118270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:04.118270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:04.118270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:04.118270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:04.118270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:04.118705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:04.118705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:04.118705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:04.118705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:04.118705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:04.118705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:04.118705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:04.118705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.118705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:04.118940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.118940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:04.118940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:04.118940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:04.118940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.118940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:04.118940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:04.118940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:22:04.118940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:04.118940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:04.118940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.118940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:04.119263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:04.119263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:04.119263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.119263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:04.119263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:04.119263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:04.119263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:04.119263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:04.119263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:04.119263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:04.119263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:04.119263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:04.119263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:04.119263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:04.119636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:04.119636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:04.119636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.119636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:04.119636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.119636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:04.119636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:04.119636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:04.119636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:04.119636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:04.119636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.119636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:04.119636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:04.119636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.120007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:04.120007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:04.120007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.120007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:04.120007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:04.120007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.120007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:04.120007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:04.120007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:22:04.120007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:04.120007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:04.120007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.120007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:04.120007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:04.120007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:04.120007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.120007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:04.120007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:04.120007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:04.120438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:04.120438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:04.120438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:04.120438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:04.120438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:04.120438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:04.120438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:04.120438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:04.120438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:04.120438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:04.120438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.120438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:04.134165 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:04.134409 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:04.134499 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:04.134616 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:04.135094 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:04.135330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.135775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:04.135775 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:04.135775 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:04.135775 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:04.135775 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:04.135775 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:04.135775 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.135775 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:04.136575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.136575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:04.136575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:04.136575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:04.136575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.136575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:04.136575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:04.136575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:04.136575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.136575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:04.136575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:04.136575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:04.136575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:04.136575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:04.136575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:04.136932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:04.136932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:04.136932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:04.136932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:04.136932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:04.136932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:04.136932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:04.136932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.136932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:04.137359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.137359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:04.137359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:04.137359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:04.137359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.137359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:04.137359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:04.137359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:04.137359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:04.137359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.137359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:04.137908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:04.137908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:04.137908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.137908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:04.137908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:04.137908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:04.137908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:04.137908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:04.137908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:04.137908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:04.137908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:04.137908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:04.137908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:04.137908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:04.138605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:04.138605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:04.138605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.138605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:04.138605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.138605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:04.138605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:04.138605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:04.138605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:04.138605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:04.138605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.138605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:04.138605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:04.138605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.139106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:04.139106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:04.139106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.139106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:04.139106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:04.139106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.139106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:04.139106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:04.139106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:04.139106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:04.139106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.139106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:04.139106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:04.139106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:04.139106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.139106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:04.139106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:04.139106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:04.139106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:04.139424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:04.139424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:04.139424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:04.139424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:04.139424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:04.139424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:04.139424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:04.139424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:04.139424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:04.139424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.139424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:04.139424 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:04.139424 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:04.139655 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:04.139655 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:04.139655 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:04.139655 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:04.139655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.140079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:04.140079 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:04.140079 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:04.140079 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:04.140079 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:04.140079 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:04.140079 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.140079 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:04.140642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.140642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:04.140642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:04.140642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:04.140642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.140642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:04.140642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:04.140642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.140642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:04.140642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:04.140642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:04.140642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.141102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:04.141102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:04.141102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:04.141102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:04.141102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:04.141102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:04.141102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:04.141102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:04.141102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:04.141102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:04.141102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:04.141102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:04.141102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:04.141555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.141555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:04.141555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.141555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:04.141555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:04.141555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:04.141555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:04.141555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:04.141555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.141555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:04.141555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:04.141555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.141555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:04.141555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:04.141555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.142199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:04.142199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:04.142199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:04.142199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:04.142199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:04.142199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.142199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:04.142199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:04.142199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:04.142199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:04.142199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:04.142199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:04.142199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:04.142199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:04.142199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:04.142199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:04.142199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:04.142199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:04.142199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:04.142721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:04.142721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:04.142721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:04.142721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:04.142721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:04.142721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:08.583762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:08.583762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:08.583762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:08.583762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:08.628100 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:08.628335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:08.628860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:08.628860 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:08.628860 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:08.628860 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:08.628860 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:08.628860 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:08.628860 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:08.628860 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:08.629629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:08.629629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:08.629629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:08.629629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:08.629629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:08.629629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:08.629629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:08.629629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:08.629629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:08.629629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:08.629629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:08.629629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:08.629874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:08.629874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:08.629874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:08.629874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:08.629874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:08.629874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:08.629874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:08.629874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:08.629874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:08.629874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:08.629874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:08.629874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:08.629874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:08.630074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:08.630074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:08.630398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:08.630398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:08.630398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:08.630398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:08.630398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:08.630398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:08.630398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:08.630398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:08.630398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:08.630398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:08.630398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:08.630398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:08.630398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:08.630398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:08.630398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:08.630398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:08.630731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:08.630731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:08.630731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:08.630731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:08.630731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:08.630731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:08.630731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:08.630731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:08.630731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:08.630731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:08.630731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:08.630731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:08.630731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:08.630731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:08.630731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:08.630731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:08.630731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:08.630731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:08.630731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:08.631050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:08.631050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:08.631050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:09.104009 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:09.104332 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:09.104449 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:09.104943 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:09.105151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.105716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:09.105716 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:09.105716 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:09.105716 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:09.105716 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:09.105716 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:09.105716 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.105716 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:09.106453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.106453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:09.106453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:09.106453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:09.106453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.106453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:09.106453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:09.106453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.106453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:09.106453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:09.106453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:09.106453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.106740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:09.106740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:09.106740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:09.106740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:09.106740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:09.106740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:09.106740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:09.106740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:09.106740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:09.106740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:09.106740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:09.106740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:09.106740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:09.106972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.106972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:09.107355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.107355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:09.107355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:09.107355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:09.107355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:09.107355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:09.107355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.107355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:09.107355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:09.107355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.107355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:09.107355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:09.107355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.107355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:09.107355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:09.107355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.108055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:09.108055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:09.108055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.108055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:09.108055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:09.108055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:09.108055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.108055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:09.108055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:09.108055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:09.108055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:09.108055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:09.108055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:09.108055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:09.108055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:09.108055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:09.108055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:09.108055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:09.108055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:09.108374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:09.108374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.108374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:09.116404 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:09.116545 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:09.116625 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:09.116712 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:22:09.117133 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:09.117363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.117854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:09.117854 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:09.117854 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:09.117854 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:09.117854 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:09.117854 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:09.117854 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.117854 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:09.118394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.118394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:09.118394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:09.118394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:09.118394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.118394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:09.118394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:09.118394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:09.118394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.118394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:09.118394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:09.118394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:09.118394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:09.118394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:09.118394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:09.118647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:09.118647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:09.118647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:09.118647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:09.118647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:09.118647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:09.118647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:09.118647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.118647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:09.118979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.118979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:09.118979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:09.118979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:09.118979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.118979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:09.118979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:09.118979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:22:09.118979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:09.118979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:09.118979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.118979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:09.119609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:09.119609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:09.119609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.119609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:09.119609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:09.119609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:09.119609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:09.119609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:09.119609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:09.119609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:09.119609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:09.119609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:09.119609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:09.119609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:09.119912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:09.119912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:09.119912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.119912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:09.119912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.119912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:09.119912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:09.119912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:09.119912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:09.119912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:09.119912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.119912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:09.119912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:09.119912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.120118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:09.120118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:09.120118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.120118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:09.120118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:09.120118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.120118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:09.120118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:09.120118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:22:09.120118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:09.120118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:09.120118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.120118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:09.120118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:09.120118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:09.120118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.120118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:09.120118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:09.120118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:09.120428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:09.120428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:09.120428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:09.120428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:09.120428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:09.120428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:09.120428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:09.120428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:09.120428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:09.120428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:09.120428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.120428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:09.134441 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:09.134577 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:09.134661 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:09.134783 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:09.135169 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:09.135386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.135773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:09.135773 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:09.135773 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:09.135773 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:09.135773 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:09.135773 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:09.135773 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.135773 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:09.136356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.136356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:09.136356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:09.136356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:09.136356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.136356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:09.136356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:09.136356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:09.136356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.136356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:09.136356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:09.136356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:09.136356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:09.136356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:09.136356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:09.136586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:09.136586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:09.136586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:09.136586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:09.136586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:09.136586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:09.136586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:09.136586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.136586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:09.136928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.136928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:09.136928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:09.136928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:09.136928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.136928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:09.136928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:09.136928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:09.136928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:09.136928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.136928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:09.137106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:09.137106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:09.137106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.137106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:09.137106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:09.137106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:09.137106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:09.137106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:09.137106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:09.137106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:09.137106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:09.137106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:09.137106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:09.137106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:09.137338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:09.137338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:09.137338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.137338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:09.137681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.137681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:09.137681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:09.137681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:09.137681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:09.137681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:09.137681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.137681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:09.137681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:09.137681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.137681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:09.137681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:09.137681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.137681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:09.137681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:09.137681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.137681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:09.138012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:09.138012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:09.138012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:09.138012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.138012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:09.138012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:09.138012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:09.138012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.138012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:09.138012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:09.138012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:09.138012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:09.138012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:09.138012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:09.138012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:09.138012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:09.138012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:09.138012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:09.138012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:09.138306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:09.138306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:09.138306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.138306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:09.138925 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:09.139038 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:09.139122 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:09.139264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:09.139537 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:09.139715 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:09.139804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.140187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:09.140187 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:09.140187 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:09.140187 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:09.140187 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:09.140187 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:09.140187 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.140187 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:09.140744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.140744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:09.140744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:09.140744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:09.140744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.140744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:09.140744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:09.140744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.140744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:09.140744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:09.140744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:09.140744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.140947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:09.140947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:09.140947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:09.140947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:09.140947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:09.140947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:09.140947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:09.140947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:09.140947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:09.140947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:09.140947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:09.140947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:09.140947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:09.141146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.141146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:09.141420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.141420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:09.141420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:09.141420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:09.141420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:09.141420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:09.141420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.141420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:09.141420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:09.141420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.141420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:09.141420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:09.141420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.141420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:09.141420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:09.141420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:09.141727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:09.141727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:09.141727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.141727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:09.141727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:09.141727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:09.141727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:09.141727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:09.141727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:09.141727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:09.141727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:09.141727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:09.141727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:09.141727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:09.141727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:09.141727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:09.141727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:09.141727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:09.141727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:09.142011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:09.142011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:09.142011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:13.582796 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:13.583457 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:13.583457 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:13.583627 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:13.607921 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:13.608122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:13.608654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:13.608654 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:13.608654 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:13.608654 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:13.608654 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:13.608654 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:13.608654 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:13.608654 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:13.609350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:13.609350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:13.609350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:13.609350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:13.609350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:13.609350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:13.609350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:13.609350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:13.609350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:13.609350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:13.609350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:13.609350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:13.609720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:13.609720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:13.609720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:13.609720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:13.609720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:13.609720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:13.609720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:13.609720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:13.609720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:13.609720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:13.609720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:13.609720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:13.609720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:13.610060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:13.610060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:13.610060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:13.610060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:13.610060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:13.610060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:13.610060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:13.610060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:13.610060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:13.610060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:13.610060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:13.610060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:13.610060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:13.610060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:13.610060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:13.610462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:13.610462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:13.610462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:13.610462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:13.610462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:13.610462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:13.610462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:13.610462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:13.610462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:13.610462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:13.610462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:13.610462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:13.610462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:13.610462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:13.610462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:13.610462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:13.610462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:13.610462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:13.610462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:13.610935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:13.610935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:13.610935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:13.610935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:13.610935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:13.610935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:14.104443 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:14.104716 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:14.104716 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:14.105448 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:14.105671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.106153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:14.106153 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:14.106153 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:14.106153 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:14.106153 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:14.106153 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:14.106153 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.106153 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:14.106928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.106928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:14.106928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:14.106928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:14.106928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.106928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:14.106928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:14.106928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.106928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:14.106928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:14.106928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:14.106928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.107193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:14.107193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:14.107193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:14.107193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:14.107193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:14.107193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:14.107193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:14.107193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:14.107193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:14.107193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:14.107193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:14.107193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:14.107193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:14.107423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.107423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:14.107679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.107679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:14.107679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:14.107679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:14.107679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:14.107679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:14.107679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.107679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:14.107679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:14.107679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.107679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:14.107679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:14.107679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.107679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:14.107679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:14.107679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.107978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:14.107978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:14.107978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.107978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:14.107978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:14.107978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:14.107978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.107978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:14.107978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:14.107978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:14.107978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:14.107978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:14.107978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:14.107978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:14.107978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:14.107978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:14.107978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:14.107978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:14.107978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:14.108319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:14.108319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.108319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:14.116710 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:14.116914 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:14.117005 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:14.117091 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:22:14.117511 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:14.117696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.118141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:14.118141 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:14.118141 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:14.118141 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:14.118141 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:14.118141 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:14.118141 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.118141 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:14.118700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.118700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:14.118700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:14.118700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:14.118700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.118700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:14.118700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:14.118700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:14.118700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.118700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:14.118700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:14.118700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:14.118700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:14.118700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:14.118700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:14.119002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:14.119002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:14.119002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:14.119002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:14.119002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:14.119002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:14.119002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:14.119002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.119002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:14.119400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.119400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:14.119400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:14.119400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:14.119400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.119400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:14.119400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:14.119400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:22:14.119400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:14.119400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:14.119400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.119400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:14.119863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:14.119863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:14.119863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.119863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:14.119863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:14.119863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:14.119863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:14.119863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:14.119863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:14.119863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:14.119863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:14.119863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:14.119863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:14.119863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:14.120380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:14.120380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:14.120380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.120380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:14.120380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.120380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:14.120380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:14.120380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:14.120380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:14.120380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:14.120380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.120380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:14.120380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:14.120380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.120603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:14.120603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:14.120603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.120603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:14.120603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:14.120603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.120603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:14.120603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:14.120603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:22:14.120603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:14.120603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:14.120603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.120603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:14.120603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:14.120603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:14.120603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.120603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:14.120603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:14.120603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:14.121409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:14.121409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:14.121409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:14.121409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:14.121409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:14.121409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:14.121409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:14.121409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:14.121409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:14.121409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:14.121409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.121409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:14.134744 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:14.134922 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:14.135014 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:14.135118 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:14.135470 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:14.135668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.136078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:14.136078 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:14.136078 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:14.136078 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:14.136078 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:14.136078 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:14.136078 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.136078 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:14.136664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.136664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:14.136664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:14.136664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:14.136664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.136664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:14.136664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:14.136664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:14.136664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.136664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:14.136664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:14.136664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:14.136664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:14.136664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:14.136664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:14.136931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:14.136931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:14.136931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:14.136931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:14.136931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:14.136931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:14.136931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:14.136931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.136931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:14.137394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.137394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:14.137394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:14.137394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:14.137394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.137394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:14.137394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:14.137394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:14.137394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:14.137394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.137394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:14.137605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:14.137605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:14.137605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.137605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:14.137605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:14.137605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:14.137605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:14.137605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:14.137605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:14.137605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:14.137605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:14.137605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:14.137605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:14.137605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:14.137807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:14.137807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:14.137807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.137807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:14.138352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.138352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:14.138352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:14.138352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:14.138352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:14.138352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:14.138352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.138352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:14.138352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:14.138352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.138352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:14.138352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:14.138352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.138352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:14.138352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:14.138352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.138352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:14.138717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:14.138717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:14.138717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:14.138717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.138717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:14.138717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:14.138717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:14.138717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.138717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:14.138717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:14.138717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:14.138717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:14.138717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:14.138717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:14.138717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:14.138717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:14.138717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:14.138717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:14.138717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:14.139037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:14.139037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:14.139037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.139037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:14.139524 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:14.139524 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:14.139643 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:14.139689 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:14.139948 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:14.140110 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:14.140193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.140564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:14.140564 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:14.140564 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:14.140564 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:14.140564 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:14.140564 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:14.140564 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.140564 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:14.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:14.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:14.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:14.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:14.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:14.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:14.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:14.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:14.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.141743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:14.141743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:14.141743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:14.141743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:14.141743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:14.141743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:14.141743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:14.141743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:14.141743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:14.141743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:14.141743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:14.141743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:14.141743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:14.142201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.142201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:14.142368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.142368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:14.142368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:14.142368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:14.142368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:14.142368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:14.142368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.142368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:14.142368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:14.142368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.142368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:14.142368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:14.142368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.142368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:14.142368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:14.142368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:14.142695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:14.142695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:14.142695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.142695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:14.142695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:14.142695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:14.142695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:14.142695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:14.142695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:14.142695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:14.142695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:14.142695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:14.142695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:14.142695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:14.142695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:14.142695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:14.142695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:14.142695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:14.142695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:14.143006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:14.143006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:14.143006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:18.583093 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:18.583615 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:18.583615 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:18.584150 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:18.607022 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:18.607233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:18.607795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:18.607795 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:18.607795 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:18.607795 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:18.607795 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:18.607795 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:18.607795 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:18.607795 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:18.608513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.024 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:18.608513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:18.608513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:18.608513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:18.608513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:18.608513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:18.608513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:18.608513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:18.608513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:18.608513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:18.608513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:18.608513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:18.608726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:18.608726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:18.608726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:18.608726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:18.608726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:18.608726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:18.608726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:18.608726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:18.608726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:18.608726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:18.608726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:18.608726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:18.608726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:18.608947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:18.608947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:18.609239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:18.609239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:18.609239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:18.609239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:18.609239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:18.609239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:18.609239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:18.609239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:18.609239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:18.609239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:18.609239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:18.609239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:18.609239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:18.609239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:18.609239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:18.609239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:18.609514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:18.609514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:18.609514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:18.609514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:18.609514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:18.609514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:18.609514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:18.609514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:18.609514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:18.609514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:18.609514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:18.609514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:18.609514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:18.609514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:18.609514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:18.609514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:18.609514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:18.609514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:18.609514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:18.609789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:18.609789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:18.609789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:19.104865 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:19.105221 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:19.105385 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:19.105969 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:19.106200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.106758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:19.106758 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:19.106758 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:19.106758 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:19.106758 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:19.106758 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:19.106758 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.106758 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:19.107457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.107457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:19.107457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:19.107457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:19.107457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.107457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:19.107457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:19.107457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.107457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:19.107457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:19.107457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:19.107457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.107895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:19.107895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:19.107895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:19.107895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:19.107895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:19.107895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:19.107895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:19.107895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:19.107895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:19.107895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:19.107895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:19.107895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:19.107895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:19.108277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.108277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:19.108682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.108682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:19.108682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:19.108682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:19.108682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:19.108682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:19.108682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.108682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:19.108682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:19.108682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.108682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:19.108682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:19.108682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.108682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:19.108682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:19.108682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.109960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:19.109960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:19.109960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.109960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:19.109960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:19.109960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:19.109960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.109960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:19.109960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:19.109960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:19.109960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:19.109960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:19.109960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:19.109960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:19.109960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:19.109960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:19.109960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:19.109960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:19.109960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:19.110735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:19.110735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.110735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:19.117047 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:19.117358 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:19.117480 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:19.117589 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:22:19.117953 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:19.118118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.118545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:19.118545 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:19.118545 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:19.118545 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:19.118545 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:19.118545 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:19.118545 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.118545 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:19.119135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.119135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:19.119135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:19.119135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:19.119135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.119135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:19.119135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:19.119135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:19.119135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.119135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:19.119135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:19.119135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:19.119135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:19.119135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:19.119135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:19.119602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:19.119602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:19.119602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:19.119602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:19.119602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:19.119602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:19.119602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:19.119602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.119602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:19.119850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.119850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:19.119850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:19.119850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:19.119850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.119850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:19.119850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:19.119850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:22:19.119850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:19.119850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:19.119850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.119850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:19.120178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:19.120178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:19.120178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.120178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:19.120178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:19.120178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:19.120178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:19.120178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:19.120178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:19.120178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:19.120178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:19.120178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:19.120178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:19.120178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:19.120528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:19.120528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:19.120528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.120528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:19.120528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.120528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:19.120528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:19.120528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:19.120528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:19.120528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:19.120528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.120528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:19.120528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:19.120528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.120912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:19.120912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:19.120912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.120912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:19.120912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:19.120912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.120912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:19.120912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:19.120912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:22:19.120912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:19.120912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:19.120912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.120912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:19.120912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:19.120912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:19.120912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.120912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:19.120912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:19.120912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:19.121429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:19.121429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:19.121429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:19.121429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:19.121429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:19.121429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:19.121429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:19.121429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:19.121429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:19.121429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:19.121429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.121429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:19.135065 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:19.135395 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:19.135485 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:19.135605 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:19.135902 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:19.136098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.136515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:19.136515 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:19.136515 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:19.136515 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:19.136515 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:19.136515 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:19.136515 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.136515 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:19.137083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.137083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:19.137083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:19.137083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:19.137083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.137083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:19.137083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:19.137083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:19.137083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.137083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:19.137083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:19.137083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:19.137083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:19.137083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:19.137083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:19.137509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:19.137509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:19.137509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:19.137509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:19.137509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:19.137509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:19.137509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:19.137509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.137509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:19.137740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.137740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:19.137740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:19.137740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:19.137740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.137740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:19.137740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:19.137740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:19.137740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:19.137740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.137740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:19.138055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:19.138055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:19.138055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.138055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:19.138055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:19.138055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:19.138055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:19.138055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:19.138055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:19.138055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:19.138055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:19.138055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:19.138055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:19.138055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:19.138414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:19.138414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:19.138414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.138414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:19.138414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.138414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:19.138414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:19.138414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:19.138414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:19.138414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:19.138414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.138414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:19.138414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:19.138414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.138786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:19.138786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:19.138786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.138786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:19.138786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:19.138786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.138786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:19.138786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:19.138786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:19.138786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:19.138786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.138786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:19.138786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:19.138786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:19.138786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.138786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:19.138786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:19.138786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:19.138786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:19.139296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:19.139296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:19.139296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:19.139296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:19.139296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:19.139296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:19.139296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:19.139296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:19.139296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:19.139296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.139296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:19.139587 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:19.140618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:19.140618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:19.140618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:19.140618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:19.140618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:19.140618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.141100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:19.141100 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:19.141100 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:19.141100 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:19.141100 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:19.141100 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:19.141100 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.141100 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:19.141696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.141696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:19.141696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:19.141696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:19.141696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.141696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:19.141696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:19.141696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.141696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:19.141696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:19.141696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:19.141696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.142564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:19.142564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:19.142564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:19.142564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:19.142564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:19.142564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:19.142564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:19.142564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:19.142564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:19.142564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:19.142564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:19.142564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:19.142564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:19.143561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.143561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:19.143561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.143561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:19.143561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:19.143561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:19.143561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:19.143561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:19.143561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.143561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:19.143561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:19.143561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.143561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:19.143561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:19.143561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.144267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:19.144267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:19.144267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:19.144267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:19.144267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:19.144267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.144267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:19.144267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:19.144267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:19.144267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:19.144267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:19.144267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:19.144267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:19.144267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:19.144267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:19.144267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:19.144267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:19.144267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:19.144267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:19.144750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:19.144750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:19.144750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:19.144750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:19.144750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:19.144750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:23.583536 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:23.584012 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:23.584012 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:23.584494 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:23.607613 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:23.607767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:23.608180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:23.608180 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:23.608180 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:23.608180 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:23.608180 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:23.608180 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:23.608180 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:23.608180 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:23.608842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.024 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:23.608842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:23.608842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:23.608842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:23.608842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:23.608842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:23.608842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:23.608842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:23.608842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:23.608842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:23.608842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:23.608842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:23.609283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:23.609283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:23.609283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:23.609283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:23.609283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:23.609283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:23.609283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:23.609283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:23.609283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:23.609283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:23.609283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:23.609283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:23.609283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:23.609489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:23.609489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:23.609532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:23.609532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:23.609532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:23.609532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:23.609532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:23.609532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:23.609532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:23.609532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:23.609532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:23.609532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:23.609532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:23.609532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:23.609532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:23.609532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:23.609532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:23.609532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:23.609778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:23.609778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:23.609778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:23.609778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:23.609778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:23.609778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:23.609778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:23.609778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:23.609778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:23.609778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:23.609778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:23.609778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:23.609778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:23.609778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:23.609778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:23.609778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:23.609778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:23.609778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:23.609778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:23.610093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:23.610093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:23.610093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:24.105281 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:24.105561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:24.105651 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:24.106058 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:24.106193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.106605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:24.106605 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:24.106605 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:24.106605 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:24.106605 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:24.106605 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:24.106605 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.106605 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:24.107228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.107228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:24.107228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:24.107228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:24.107228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.107228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:24.107228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:24.107228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.107228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:24.107228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:24.107228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:24.107228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.107553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:24.107553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:24.107553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:24.107553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:24.107553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:24.107553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:24.107553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:24.107553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:24.107553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:24.107553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:24.107553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:24.107553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:24.107553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:24.107868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.107868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:24.107868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.107868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:24.107868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:24.107868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:24.107868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:24.107868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:24.107868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.107868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:24.107868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:24.107868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.107868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:24.107868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:24.107868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.108228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:24.108228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:24.108228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.108228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:24.108228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:24.108228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.108228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:24.108228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:24.108228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:24.108228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.108228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:24.108228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:24.108228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:24.108228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:24.108228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:24.108228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:24.108228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:24.108228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:24.108228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:24.108693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:24.108693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:24.108693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:24.108693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:24.108693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.108693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:24.117297 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:24.117473 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:24.117576 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:24.117641 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:22:24.117870 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:24.118011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.118331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:24.118331 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:24.118331 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:24.118331 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:24.118331 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:24.118331 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:24.118331 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.118331 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:24.118820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.118820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:24.118820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:24.118820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:24.118820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.118820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:24.118820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:24.118820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:24.118820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.118820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:24.118820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:24.118820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:24.118820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:24.118820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:24.118820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:24.119199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:24.119199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:24.119199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:24.119199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:24.119199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:24.119199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:24.119199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:24.119199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.119199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:24.119443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.119443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:24.119443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:24.119443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:24.119443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.119443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:24.119443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:24.119443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:22:24.119443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:24.119443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:24.119443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.119443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:24.119730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:24.119730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:24.119730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.119730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:24.119730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:24.119730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:24.119730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:24.119730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:24.119730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:24.119730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:24.119730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:24.119730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:24.119730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:24.119730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:24.120131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:24.120131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:24.120131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.120131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:24.120131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.120131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:24.120131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:24.120131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:24.120131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:24.120131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:24.120131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.120131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:24.120131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:24.120131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.120462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:24.120462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:24.120462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.120462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:24.120462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:24.120462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.120462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:24.120462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:24.120462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:22:24.120462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:24.120462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:24.120462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.120462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:24.120462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:24.120462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:24.120462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.120462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:24.120462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:24.120462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:24.120894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:24.120894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:24.120894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:24.120894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:24.120894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:24.120894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:24.120894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:24.120894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:24.120894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:24.120894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:24.120894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.120894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:24.135313 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:24.135537 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:24.135537 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:24.135653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:24.135863 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:24.136039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.136359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:24.136359 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:24.136359 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:24.136359 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:24.136359 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:24.136359 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:24.136359 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.136359 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:24.136952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.136952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:24.136952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:24.136952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:24.136952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.136952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:24.136952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:24.136952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:24.136952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.136952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:24.136952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:24.136952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:24.136952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:24.136952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:24.136952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:24.137331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:24.137331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:24.137331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:24.137331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:24.137331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:24.137331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:24.137331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:24.137331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.137331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:24.137555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.137555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:24.137555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:24.137555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:24.137555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.137555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:24.137555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:24.137555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:24.137555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:24.137555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.137555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:24.137821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:24.137821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:24.137821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.137821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:24.137821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:24.137821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:24.137821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:24.137821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:24.137821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:24.137821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:24.137821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:24.137821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:24.137821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:24.137821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:24.138166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:24.138166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:24.138166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.138166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:24.138284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.138284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:24.138284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:24.138284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:24.138284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:24.138284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:24.138284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.138284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:24.138284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:24.138284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.138284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:24.138284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:24.138284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.138284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:24.138284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:24.138284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.138284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:24.138667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:24.138667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:24.138667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:24.138667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.138667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:24.138667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:24.138667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:24.138667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.138667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:24.138667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:24.138667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:24.138667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:24.138667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:24.138667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:24.138667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:24.138667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:24.138667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:24.138667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:24.138667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:24.139110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:24.139110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:24.139110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.139110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:24.140045 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:24.140045 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:24.140045 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:24.140286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:24.140372 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:24.140458 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:24.140537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.140839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:24.140839 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:24.140839 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:24.140839 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:24.140839 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:24.140839 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:24.140839 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.140839 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:24.141408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.141408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:24.141408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:24.141408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:24.141408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.141408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:24.141408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:24.141408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.141408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:24.141408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:24.141408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:24.141408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.142177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:24.142177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:24.142177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:24.142177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:24.142177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:24.142177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:24.142177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:24.142177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:24.142177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:24.142177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:24.142177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:24.142177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:24.142177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:24.143028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.143028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:24.143028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.143028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:24.143028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:24.143028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:24.143028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:24.143028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:24.143028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.143028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:24.143028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:24.143028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.143028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:24.143028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:24.143028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.143982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:24.143982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:24.143982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:24.143982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:24.143982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:24.143982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.143982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:24.143982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:24.143982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:24.143982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:24.143982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:24.143982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:24.143982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:24.143982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:24.143982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:24.143982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:24.143982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:24.143982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:24.143982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:24.144708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:24.144708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:24.144708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:24.144708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:24.144708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:24.144708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:28.583923 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:28.585938 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:28.585938 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:28.585938 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:28.608075 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:28.608295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:28.610201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:28.610201 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:28.610201 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:28.610201 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:28.610201 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:28.610201 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:28.610201 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:28.610201 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:28.610201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.024 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:28.610201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:28.610201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:28.610201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:28.610201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:28.610201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:28.610201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:28.610825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:28.610825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:28.610825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:28.610825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:28.610825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:28.610825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:28.610825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:28.610825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:28.610825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:28.610825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:28.610825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:28.610825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:28.610825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:28.610825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:28.611327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:28.611327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:28.611327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:28.611327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:28.611327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:28.611327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:28.611327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:28.611327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:28.611327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:28.611327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:28.611327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:28.611327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:28.611327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:28.611327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:28.611735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:28.612074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:28.612074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:28.612074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:28.612074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:28.612074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:28.612074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:28.612074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:28.612074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:28.612074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:29.105682 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:29.106517 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:29.106517 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:29.106663 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:29.106827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.108549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:29.108549 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:29.108549 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:29.108549 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:29.108549 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:29.108549 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:29.108549 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.108549 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:29.108549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.108549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:29.108549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:29.108549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:29.108549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.108549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:29.108549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:29.108849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.108849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:29.108849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:29.108849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:29.108849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.108849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:29.108849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:29.108849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:29.108849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:29.108849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:29.108849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:29.108849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:29.108849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:29.108849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:29.109081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:29.109081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:29.109081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:29.109081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:29.109081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.109081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:29.109081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.109081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:29.109081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:29.109081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:29.109081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:29.109081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:29.109081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.109081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:29.109320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:29.109625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:29.109625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:29.109625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:29.109625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:29.109625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:29.109625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:29.109625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:29.109625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:29.109625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.109625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:29.117539 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:29.117780 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:29.117974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:29.118141 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:22:29.118485 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:29.118626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.120707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:29.120707 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:29.120707 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:29.120707 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:29.120707 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:29.120707 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:29.120707 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.120707 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:29.120707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.120707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:29.120707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:29.120707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:29.120707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.120707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:29.120707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:29.120707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:29.120707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.120707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:29.121472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:29.121472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:29.121472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:29.121472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:29.121472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:29.121472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:29.121472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:29.121472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:29.121472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:29.121472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:29.121472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:29.121472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:29.121472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.121472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:29.121472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.122006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:29.122006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:29.122006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:29.122006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.122006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:29.122006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:29.122006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:22:29.122006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:29.122006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:29.122006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.122006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:29.122006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:29.122006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:29.122489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.122489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:29.122489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:29.122489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:29.122489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:29.122489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:29.122489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:29.122489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:29.122489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:29.122489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:29.122489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:29.122489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:29.122489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:29.122489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:29.122977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.122977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:29.122977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.122977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:29.122977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:29.122977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:29.122977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:29.122977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:29.122977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.122977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:29.122977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:29.122977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.122977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:29.122977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:29.122977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.123534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:29.123534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:29.123534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.123534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:29.123534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:29.123534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:22:29.123534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:29.123534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:29.123534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.123534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:29.123534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:29.123534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:29.123534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.123534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:29.123534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:29.123534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:29.123534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:29.123534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:29.123534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:29.123892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:29.123892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:29.123892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:29.123892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:29.123892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:29.123892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:29.123892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:29.123892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.123892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:29.135598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:29.135910 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:29.136099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:29.136335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:29.136704 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:29.136961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.139187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:29.139187 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:29.139187 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:29.139187 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:29.139187 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:29.139187 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:29.139187 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.139187 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:29.139187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.139187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:29.139187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:29.139187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:29.139187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.139187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:29.139187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:29.139187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:29.139187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.139187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:29.139878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:29.139878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:29.139878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:29.139878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:29.139878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:29.139878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:29.139878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:29.139878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:29.139878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:29.139878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:29.139878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:29.139878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:29.139878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.139878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:29.139878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.140540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:29.140540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:29.140540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:29.140540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.140540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:29.140540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:29.140540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:29.140540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:29.140540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.140540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:29.140540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:29.140540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:29.140540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.141027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:29.141027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:29.141027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:29.141027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:29.141027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:29.141027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:29.141027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:29.141027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:29.141027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:29.141027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:29.141027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:29.141027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:29.141027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:29.141478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.141478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:29.141478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.141478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:29.141478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:29.141478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:29.141478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:29.141478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:29.141478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.141478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:29.141478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:29.141478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.141478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:29.141478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:29.141478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.142709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:29.142709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:29.142709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.142709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:29.142709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:29.142709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:29.142709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:29.142709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.142709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:29.142709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:29.142709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:29.142709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.142709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:29.142709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:29.142709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:29.142709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:29.142709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:29.142709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:29.142709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:29.144583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:29.144583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:29.144583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:29.144583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:29.144583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:29.144583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:29.144583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.144583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:29.144583 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:29.144583 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:29.144583 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:29.144783 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:29.144783 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:29.144783 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:29.144783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.144783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:29.144783 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:29.144783 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:29.144783 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:29.144783 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:29.144783 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:29.144783 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.144783 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:29.145021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.145021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:29.145021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:29.145021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:29.145021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.145021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:29.145021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:29.145021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.145021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:29.145021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:29.145021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:29.145021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.145234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:29.145234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:29.145234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:29.145234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:29.145234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:29.145234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:29.145234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:29.145234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:29.145234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:29.145234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:29.145234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:29.145234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:29.145234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:29.145577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.145577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:29.145631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.145631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:29.145631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:29.145631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:29.145631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:29.145631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:29.145631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.145631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:29.145631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:29.145631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.145631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:29.145631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:29.145631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.145631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:29.145631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:29.145631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:29.145886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:29.145886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:29.145886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.145886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:29.145886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:29.145886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:29.145886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:29.145886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:29.145886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:29.145886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:29.145886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:29.145886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:29.145886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:29.145886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:29.145886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:29.145886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:29.145886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:29.145886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:29.145886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:29.146185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:29.146185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:29.146185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:33.584905 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:33.585380 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:33.585380 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:33.585993 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:33.612716 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:33.612935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:33.613429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:33.613429 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:33.613429 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:33.613429 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:33.613429 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:33.613429 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:33.613429 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:33.613429 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:33.615099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:33.615099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:33.615099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:33.615099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:33.615099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:33.615099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:33.615099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:33.615099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:33.615099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:33.615099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:33.615099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:33.615099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:33.615374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:33.615374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:33.615374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:33.615374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:33.615374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:33.615374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:33.615374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:33.615374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:33.615374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:33.615374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:33.615374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:33.615374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:33.615374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:33.615629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:33.615629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:33.615629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:33.615629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:33.615629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:33.615629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:33.615629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:33.615629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:33.615629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:33.615629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:33.615629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:33.615629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:33.615629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:33.615629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:33.615629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:33.615904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:33.615904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:33.615904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:33.615904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:33.615904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:33.615904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:33.615904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:33.615904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:33.615904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:33.615904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:33.615904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:33.615904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:33.615904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:33.615904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:33.615904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:33.615904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:33.615904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:33.615904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:33.615904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:33.616227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:33.616227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:33.616227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:33.616227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:33.616227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:33.616227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:34.106163 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:34.106736 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:34.106858 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:34.107361 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:34.107500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.108085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:34.108085 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:34.108085 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:34.108085 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:34.108085 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:34.108085 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:34.108085 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.108085 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:34.108855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.108855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:34.108855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:34.108855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:34.108855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.108855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:34.108855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:34.108855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.108855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:34.108855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:34.108855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:34.108855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.109120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:34.109120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:34.109120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:34.109120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:34.109120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:34.109120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:34.109120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:34.109120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:34.109120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:34.109120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:34.109120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:34.109120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:34.109120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:34.109358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.109358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:34.109621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.109621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:34.109621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:34.109621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:34.109621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:34.109621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:34.109621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.109621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:34.109621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:34.109621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.109621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:34.109621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:34.109621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.109621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:34.109621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:34.109621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.110055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:34.110055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:34.110055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.110055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:34.110055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:34.110055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:34.110055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.110055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:34.110055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:34.110055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:34.110055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:34.110055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:34.110055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:34.110055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:34.110055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:34.110055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:34.110055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:34.110055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:34.110055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:34.110427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:34.110427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.110427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:34.118019 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:34.118323 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:34.118417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:34.118509 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:22:34.118872 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:34.119054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.119471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:34.119471 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:34.119471 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:34.119471 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:34.119471 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:34.119471 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:34.119471 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.119471 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:34.120078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.120078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:34.120078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:34.120078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:34.120078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.120078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:34.120078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:34.120078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:34.120078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.120078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:34.120078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:34.120078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:34.120078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:34.120078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:34.120078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:34.120558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:34.120558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:34.120558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:34.120558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:34.120558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:34.120558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:34.120558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:34.120558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.120558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:34.120825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.120825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:34.120825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:34.120825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:34.120825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.120825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:34.120825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:34.120825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:22:34.120825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:34.120825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:34.120825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.120825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:34.121200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:34.121200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:34.121200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.121200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:34.121200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:34.121200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:34.121200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:34.121200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:34.121200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:34.121200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:34.121200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:34.121200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:34.121200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:34.121200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:34.121604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:34.121604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:34.121604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.121604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:34.121604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.121604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:34.121604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:34.121604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:34.121604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:34.121604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:34.121604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.121604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:34.121604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:34.121604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.122010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:34.122010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:34.122010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.122010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:34.122010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:34.122010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.122010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:34.122010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:34.122010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:22:34.122010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:34.122010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:34.122010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.122010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:34.122010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:34.122010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:34.122010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.122010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:34.122010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:34.122010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:34.122552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:34.122552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:34.122552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:34.122552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:34.122552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:34.122552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:34.122552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:34.122552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:34.122552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:34.122552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:34.122552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.122552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:34.136056 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:34.136392 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:34.136515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:34.136624 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:34.136998 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:34.137183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.137596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:34.137596 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:34.137596 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:34.137596 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:34.137596 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:34.137596 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:34.137596 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.137596 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:34.138180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.138180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:34.138180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:34.138180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:34.138180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.138180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:34.138180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:34.138180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:34.138180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.138180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:34.138180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:34.138180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:34.138180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:34.138180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:34.138180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:34.138611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:34.138611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:34.138611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:34.138611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:34.138611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:34.138611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:34.138611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:34.138611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.138611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:34.138891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.138891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:34.138891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:34.138891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:34.138891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.138891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:34.138891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:34.138891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:34.138891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:34.138891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.138891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:34.139208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:34.139208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:34.139208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.139208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:34.139208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:34.139208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:34.139208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:34.139208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:34.139208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:34.139208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:34.139208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:34.139208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:34.139208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:34.139208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:34.139604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:34.139604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:34.139604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.139604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:34.139604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.139604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:34.139604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:34.139604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:34.139604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:34.139604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:34.139604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.139604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:34.139604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:34.139604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.140009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:34.140009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:34.140009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.140009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:34.140009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:34.140009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.140009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:34.140009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:34.140009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:34.140009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:34.140009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.140009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:34.140009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:34.140009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:34.140009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.140009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:34.140009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:34.140009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:34.140009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:34.140552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:34.140552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:34.140552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:34.140552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:34.140552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:34.140552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:34.140552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:34.140552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:34.140552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:34.140552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.140552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:34.141796 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:34.142004 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:34.142088 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:34.142203 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:34.142478 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:34.142674 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:34.142765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.143150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:34.143150 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:34.143150 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:34.143150 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:34.143150 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:34.143150 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:34.143150 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.143150 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:34.143719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.143719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:34.143719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:34.143719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:34.143719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.143719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:34.143719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:34.143719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.143719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:34.143719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:34.143719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:34.143719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.144065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:34.144065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:34.144065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:34.144065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:34.144065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:34.144065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:34.144065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:34.144065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:34.144065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:34.144065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:34.144065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:34.144065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:34.144065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:34.144468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.144468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:34.144468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.144468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:34.144468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:34.144468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:34.144468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:34.144468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:34.144468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.144468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:34.144468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:34.144468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.144468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:34.144468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:34.144468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.144877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:34.144877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:34.144877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:34.144877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:34.144877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:34.144877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.144877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:34.144877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:34.144877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:34.144877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:34.144877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:34.144877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:34.144877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:34.144877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:34.144877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:34.144877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:34.144877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:34.144877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:34.144877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:34.145429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:34.145429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:34.145429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:34.145429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:34.145429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:34.145429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:38.585384 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:38.585896 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:38.585896 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:38.586227 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:38.609108 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:38.609325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:38.609763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:38.609763 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:38.609763 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:38.609763 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:38.609763 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:38.609763 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:38.609763 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:38.609763 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:38.610514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.024 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:38.610514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:38.610514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:38.610514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:38.610514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:38.610514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:38.610514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:38.610514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:38.610514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:38.610514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:38.610514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:38.610514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:38.610801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:38.610801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:38.610801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:38.610801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:38.610801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:38.610801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:38.610801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:38.610801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:38.610801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:38.610801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:38.610801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:38.610801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:38.610801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:38.611015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:38.611015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:38.611330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:38.611330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:38.611330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:38.611330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:38.611330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:38.611330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:38.611330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:38.611330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:38.611330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:38.611330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:38.611330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:38.611330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:38.611330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:38.611330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:38.611330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:38.611330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:38.611628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:38.611628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:38.611628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:38.611628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:38.611628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:38.611628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:38.611628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:38.611628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:38.611628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:38.611628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:38.611628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:38.611628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:38.611628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:38.611628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:38.611628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:38.611628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:38.611628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:38.611628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:38.611628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:38.611930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:38.611930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:38.611930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:39.106808 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:39.107129 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:39.107224 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:39.107760 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:39.107943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.108396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:39.108396 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:39.108396 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:39.108396 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:39.108396 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:39.108396 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:39.108396 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.108396 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:39.109097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.109097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:39.109097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:39.109097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:39.109097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.109097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:39.109097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:39.109097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.109097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:39.109097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:39.109097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:39.109097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.109343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:39.109343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:39.109343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:39.109343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:39.109343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:39.109343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:39.109343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:39.109343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:39.109343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:39.109343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:39.109343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:39.109343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:39.109343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:39.109535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.109535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:39.109807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.109807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:39.109807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:39.109807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:39.109807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:39.109807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:39.109807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.109807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:39.109807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:39.109807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.109807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:39.109807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:39.109807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.109807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:39.109807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:39.109807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.110129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:39.110129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:39.110129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.110129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:39.110129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:39.110129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:39.110129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.110129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:39.110129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:39.110129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:39.110129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:39.110129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:39.110129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:39.110129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:39.110129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:39.110129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:39.110129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:39.110129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:39.110129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:39.110524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:39.110524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.110524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:39.118403 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:39.118579 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:39.118663 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:39.118755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:22:39.119083 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:39.119247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.119663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:39.119663 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:39.119663 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:39.119663 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:39.119663 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:39.119663 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:39.119663 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.119663 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:39.120241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.120241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:39.120241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:39.120241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:39.120241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.120241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:39.120241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:39.120241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:39.120241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.120241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:39.120241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:39.120241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:39.120241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:39.120241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:39.120241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:39.120524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:39.120524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:39.120524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:39.120524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:39.120524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:39.120524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:39.120524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:39.120524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.120524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:39.120794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.120794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:39.120794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:39.120794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:39.120794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.120794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:39.120794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:39.120794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:22:39.120794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:39.120794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:39.120794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.120794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:39.121023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:39.121023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:39.121023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.121023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:39.121023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:39.121023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:39.121023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:39.121023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:39.121023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:39.121023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:39.121023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:39.121023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:39.121023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:39.121023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:39.121234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:39.121234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:39.121234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.121234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:39.121489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.121489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:39.121489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:39.121489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:39.121489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:39.121489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:39.121489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.121489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:39.121489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:39.121489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.121489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:39.121489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:39.121489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.121489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:39.121489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:39.121489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.121817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:39.121817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:39.121817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:22:39.121817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:39.121817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:39.121817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.121817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:39.121817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:39.121817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:39.121817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.121817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:39.121817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:39.121817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:39.121817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:39.121817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:39.121817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:39.121817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:39.121817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:39.122145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:39.122145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:39.122145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:39.122145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:39.122145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:39.122145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.122145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:39.136426 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:39.136638 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:39.136729 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:39.136828 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:39.137118 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:39.137317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.137687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:39.137687 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:39.137687 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:39.137687 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:39.137687 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:39.137687 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:39.137687 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.137687 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:39.138226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.138226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:39.138226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:39.138226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:39.138226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.138226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:39.138226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:39.138226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:39.138226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.138226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:39.138226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:39.138226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:39.138226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:39.138226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:39.138226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:39.138520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:39.138520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:39.138520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:39.138520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:39.138520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:39.138520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:39.138520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:39.138520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.138520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:39.138807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.138807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:39.138807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:39.138807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:39.138807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.138807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:39.138807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:39.138807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:39.138807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:39.138807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.138807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:39.139042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:39.139042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:39.139042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.139042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:39.139042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:39.139042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:39.139042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:39.139042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:39.139042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:39.139042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:39.139042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:39.139042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:39.139042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:39.139042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:39.139274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:39.139274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:39.139274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.139274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:39.139540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.139540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:39.139540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:39.139540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:39.139540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:39.139540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:39.139540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.139540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:39.139540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:39.139540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.139540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:39.139540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:39.139540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.139540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:39.139540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:39.139540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.139540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:39.139832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:39.139832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:39.139832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:39.139832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.139832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:39.139832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:39.139832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:39.139832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.139832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:39.139832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:39.139832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:39.139832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:39.139832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:39.139832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:39.139832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:39.139832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:39.139832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:39.139832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:39.139832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:39.140209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:39.140209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:39.140209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.140209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:39.142111 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:39.142264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:39.142360 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:39.142481 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:39.142732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:39.142914 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:39.142995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.143378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:39.143378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:39.143378 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:39.143378 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:39.143378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:39.143378 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:39.143378 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.143378 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:39.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:39.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:39.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:39.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:39.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:39.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:39.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:39.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:39.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.144149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:39.144149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:39.144149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:39.144149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:39.144149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:39.144149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:39.144149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:39.144149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:39.144149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:39.144149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:39.144149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:39.144149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:39.144149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:39.144372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.144372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:39.144571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.144571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:39.144571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:39.144571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:39.144571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:39.144571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:39.144571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.144571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:39.144571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:39.144571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.144571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:39.144571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:39.144571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.144571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:39.144571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:39.144571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:39.144900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:39.144900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:39.144900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.144900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:39.144900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:39.144900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:39.144900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:39.144900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:39.144900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:39.144900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:39.144900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:39.144900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:39.144900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:39.144900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:39.144900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:39.144900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:39.144900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:39.144900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:39.144900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:39.145198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:39.145198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:39.145198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:43.585734 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:43.586288 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:43.586288 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:43.586929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:43.615404 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:43.615608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:43.616208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:43.616208 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:43.616208 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:43.616208 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:43.616208 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:43.616208 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:43.616208 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:43.616208 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:43.617245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:43.617245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:43.617245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:43.617245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:43.617245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:43.617245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:43.617245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:43.617245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:43.617245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:43.617245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:43.617245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:43.617245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:43.617487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:43.617487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:43.617487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:43.617487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:43.617487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:43.617487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:43.617487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:43.617487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:43.617487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:43.617487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:43.617487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:43.617487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:43.617487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:43.617678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:43.617678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:43.618365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:43.618365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:43.618365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:43.618365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:43.618365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:43.618365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:43.618365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:43.618365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:43.618365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:43.618365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:43.618365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:43.618365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:43.618365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:43.618365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:43.618365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:43.618365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:43.618629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:43.618629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:43.618629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:43.618629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:43.618629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:43.618629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:43.618629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:43.618629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:43.618629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:43.618629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:43.618629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:43.618629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:43.618629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:43.618629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:43.618629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:43.618629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:43.618629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:43.618629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:43.618629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:43.618925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:43.618925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:43.618925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:44.107210 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:44.107533 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:44.107645 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:44.108135 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:44.108297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.109630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:44.109630 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:44.109630 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:44.109630 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:44.109630 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:44.109630 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:44.109630 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.109630 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:44.110333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.110333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:44.110333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:44.110333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:44.110333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.110333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:44.110333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:44.110333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.110333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:44.110333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:44.110333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:44.110333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.110682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:44.110682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:44.110682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:44.110682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:44.110682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:44.110682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:44.110682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:44.110682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:44.110682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:44.110682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:44.110682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:44.110682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:44.110682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:44.110945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.110945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:44.111306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.111306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:44.111306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:44.111306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:44.111306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:44.111306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:44.111306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.111306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:44.111306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:44.111306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.111306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:44.111306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:44.111306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.111306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:44.111306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:44.111306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.112106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:44.112106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:44.112106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.112106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:44.112106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:44.112106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:44.112106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.112106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:44.112106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:44.112106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:44.112106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:44.112106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:44.112106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:44.112106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:44.112106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:44.112106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:44.112106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:44.112106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:44.112106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:44.112666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:44.112666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.112666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:44.118832 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:44.119012 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:44.119093 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:44.119274 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:22:44.119467 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:44.119619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.119967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:44.119967 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:44.119967 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:44.119967 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:44.119967 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:44.119967 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:44.119967 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.119967 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:44.120477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.120477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:44.120477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:44.120477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:44.120477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.120477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:44.120477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:44.120477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:44.120477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.120477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:44.120477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:44.120477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:44.120477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:44.120477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:44.120477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:44.120723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:44.120723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:44.120723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:44.120723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:44.120723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:44.120723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:44.120723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:44.120723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.120723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:44.121053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.121053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:44.121053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:44.121053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:44.121053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.121053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:44.121053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:44.121053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:22:44.121053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:44.121053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:44.121053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.121053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:44.121638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:44.121638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:44.121638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.121638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:44.121638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:44.121638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:44.121638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:44.121638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:44.121638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:44.121638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:44.121638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:44.121638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:44.121638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:44.121638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:44.121989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:44.121989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:44.121989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.121989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:44.121989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.121989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:44.121989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:44.121989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:44.121989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:44.121989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:44.121989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.121989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:44.121989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:44.121989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.122201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:44.122201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:44.122201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.122201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:44.122201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:44.122201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.122201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:44.122201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:44.122201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:22:44.122201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:44.122201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:44.122201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.122201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:44.122201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:44.122201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:44.122201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.122201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:44.122201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:44.122201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:44.122512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:44.122512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:44.122512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:44.122512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:44.122512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:44.122512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:44.122512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:44.122512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:44.122512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:44.122512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:44.122512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.122512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:44.136834 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:44.137059 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:44.137152 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:44.137273 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:44.137554 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:44.137741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.138113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:44.138113 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:44.138113 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:44.138113 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:44.138113 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:44.138113 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:44.138113 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.138113 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:44.138615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.138615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:44.138615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:44.138615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:44.138615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.138615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:44.138615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:44.138615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:44.138615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.138615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:44.138615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:44.138615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:44.138615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:44.138615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:44.138615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:44.139037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:44.139037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:44.139037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:44.139037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:44.139037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:44.139037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:44.139037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:44.139037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.139037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:44.139322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.139322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:44.139322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:44.139322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:44.139322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.139322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:44.139322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:44.139322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:44.139322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:44.139322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.139322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:44.139562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:44.139562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:44.139562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.139562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:44.139562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:44.139562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:44.139562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:44.139562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:44.139562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:44.139562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:44.139562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:44.139562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:44.139562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:44.139562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:44.139770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:44.139770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:44.139770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.139770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:44.140156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.140156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:44.140156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:44.140156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:44.140156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:44.140156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:44.140156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.140156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:44.140156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:44.140156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.140156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:44.140156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:44.140156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.140156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:44.140156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:44.140156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.140156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:44.140721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:44.140721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:44.140721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:44.140721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.140721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:44.140721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:44.140721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:44.140721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.140721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:44.140721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:44.140721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:44.140721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:44.140721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:44.140721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:44.140721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:44.140721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:44.140721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:44.140721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:44.140721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:44.140999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:44.140999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:44.140999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.140999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:44.142483 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:44.142641 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:44.142726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:44.142858 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:44.143083 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:44.143242 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:44.143355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.144023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:44.144023 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:44.144023 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:44.144023 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:44.144023 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:44.144023 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:44.144023 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.144023 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:44.144540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.144540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:44.144540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:44.144540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:44.144540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.144540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:44.144540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:44.144540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.144540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:44.144540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:44.144540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:44.144540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.144809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:44.144809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:44.144809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:44.144809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:44.144809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:44.144809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:44.144809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:44.144809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:44.144809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:44.144809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:44.144809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:44.144809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:44.144809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:44.145027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.145027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:44.145480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.145480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:44.145480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:44.145480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:44.145480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:44.145480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:44.145480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.145480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:44.145480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:44.145480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.145480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:44.145480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:44.145480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.145480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:44.145480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:44.145480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:44.146051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:44.146051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:44.146051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.146051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:44.146051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:44.146051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:44.146051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:44.146051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:44.146051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:44.146051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:44.146051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:44.146051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:44.146051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:44.146051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:44.146051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:44.146051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:44.146051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:44.146051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:44.146051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:44.146457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:44.146457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:44.146457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:48.586123 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:48.586963 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:48.586963 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:48.587273 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:48.611022 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:48.611022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:48.611198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:48.611198 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:48.611198 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:48.611198 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:48.611198 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:48.611198 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:48.611198 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:48.611198 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:48.611864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:48.611864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:48.611864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:48.611864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:48.611864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:48.611864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:48.611864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:48.611864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:48.611864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:48.611864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:48.611864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:48.611864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:48.612484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:48.612484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:48.612484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:48.612484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:48.612484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:48.612484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:48.612484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:48.612484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:48.612484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:48.612484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:48.612484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:48.612484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:48.612484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:48.612799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:48.612799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:48.612799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:48.612799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:48.612799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:48.612799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:48.612799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:48.612799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:48.612799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:48.612799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:48.612799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:48.612799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:48.612799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:48.612799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:48.612799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:48.613200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:48.613200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:48.613200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:48.613200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:48.613200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:48.613200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:48.613200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:48.613200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:48.613200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:48.613200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:48.613200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:48.613200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:48.613200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:48.613200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:48.613200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:48.613200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:48.613200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:48.613200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:48.613200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:48.613686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:48.613686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:48.613686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:48.613686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:48.613686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:48.613686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:49.107615 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:49.107983 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:49.108100 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:49.109130 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:49.109130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.109291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:49.109291 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:49.109291 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:49.109291 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:49.109291 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:49.109291 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:49.109291 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.109291 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:49.110015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.110015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:49.110015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:49.110015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:49.110015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.110015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:49.110015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:49.110015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.110015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:49.110015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:49.110015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:49.110015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.110273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:49.110273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:49.110273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:49.110273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:49.110273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:49.110273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:49.110273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:49.110273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:49.110273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:49.110273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:49.110273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:49.110273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:49.110273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:49.110466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.110466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:49.110792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.110792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:49.110792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:49.110792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:49.110792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:49.110792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:49.110792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.110792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:49.110792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:49.110792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.110792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:49.110792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:49.110792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.110792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:49.110792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:49.110792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.111359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:49.111359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:49.111359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.111359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:49.111359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:49.111359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:49.111359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.111359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:49.111359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:49.111359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:49.111359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:49.111359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:49.111359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:49.111359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:49.111359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:49.111359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:49.111359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:49.111359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:49.111359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:49.111705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:49.111705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.111705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:49.119122 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:49.119359 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:49.119441 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:49.119533 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:22:49.119866 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:49.120149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.120668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:49.120668 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:49.120668 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:49.120668 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:49.120668 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:49.120668 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:49.120668 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.120668 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:49.121183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.121183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:49.121183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:49.121183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:49.121183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.121183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:49.121183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:49.121183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:49.121183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.121183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:49.121183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:49.121183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:49.121183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:49.121183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:49.121183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:49.121811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:49.121811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:49.121811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:49.121811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:49.121811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:49.121811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:49.121811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:49.121811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.121811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:49.121811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.121811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:49.121811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:49.121811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:49.121811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.122051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:49.122051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:49.122051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:22:49.122051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:49.122051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:49.122051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.122051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:49.122051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:49.122051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:49.122051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.122051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:49.122051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:49.122051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:49.122271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:49.122271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:49.122271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:49.122271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:49.122271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:49.122271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:49.122271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:49.122271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:49.122271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:49.122271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:49.122271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.122271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:49.122559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.122559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:49.122559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:49.122559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:49.122559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:49.122559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:49.122559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.122559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:49.122559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:49.122559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.122559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:49.122559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:49.122559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.122559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:49.122559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:49.122559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.122975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:49.122975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:49.122975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:22:49.122975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:49.122975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:49.122975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.122975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:49.122975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:49.122975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:49.122975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.122975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:49.122975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:49.122975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:49.122975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:49.122975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:49.122975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:49.122975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:49.122975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:49.123459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:49.123459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:49.123459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:49.123459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:49.123459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:49.123459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.123459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:49.137140 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:49.137397 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:49.137488 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:49.137579 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:49.138315 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:49.138315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.138637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:49.138637 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:49.138637 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:49.138637 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:49.138637 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:49.138637 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:49.138637 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.138637 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:49.139167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.139167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:49.139167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:49.139167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:49.139167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.139167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:49.139167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:49.139167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:49.139167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.139167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:49.139167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:49.139167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:49.139167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:49.139167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:49.139167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:49.139436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:49.139436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:49.139436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:49.139436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:49.139436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:49.139436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:49.139436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:49.139436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.139436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:49.139755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.139755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:49.139755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:49.139755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:49.139755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.139755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:49.139755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:49.139755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:49.139755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:49.139755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.139755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:49.140334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:49.140334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:49.140334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.140334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:49.140334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:49.140334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:49.140334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:49.140334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:49.140334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:49.140334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:49.140334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:49.140334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:49.140334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:49.140334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:49.140629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:49.140629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:49.140629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.140629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:49.140629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.140629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:49.140629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:49.140629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:49.140629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:49.140629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:49.140629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.140629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:49.140629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:49.140629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.140856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:49.140856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:49.140856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.140856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:49.140856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:49.140856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.140856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:49.140856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:49.140856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:49.140856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:49.140856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.140856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:49.140856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:49.140856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:49.140856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.140856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:49.140856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:49.140856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:49.140856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:49.141145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:49.141145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:49.141145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:49.141145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:49.141145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:49.141145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:49.141145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:49.141145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:49.141145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:49.141145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.141145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:49.142751 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:49.142936 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:49.143024 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:49.143148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:49.143449 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:49.143688 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:49.143807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.144262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:49.144262 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:49.144262 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:49.144262 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:49.144262 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:49.144262 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:49.144262 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.144262 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:49.144806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.144806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:49.144806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:49.144806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:49.144806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.144806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:49.144806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:49.144806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.144806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:49.144806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:49.144806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:49.144806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.145042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:49.145042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:49.145042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:49.145042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:49.145042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:49.145042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:49.145042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:49.145042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:49.145042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:49.145042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:49.145042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:49.145042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:49.145042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:49.145229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.145229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:49.145487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.145487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:49.145487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:49.145487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:49.145487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:49.145487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:49.145487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.145487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:49.145487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:49.145487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.145487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:49.145487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:49.145487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.145487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:49.145487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:49.145487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:49.146274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:49.146274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:49.146274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.146274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:49.146274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:49.146274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:49.146274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:49.146274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:49.146274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:49.146274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:49.146274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:49.146274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:49.146274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:49.146274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:49.146274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:49.146274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:49.146274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:49.146274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:49.146274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:49.146717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:49.146717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:49.146717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:53.586592 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:53.587174 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:53.587174 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:53.587663 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:53.588821 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=260 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:22:53.611304 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:53.611548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:53.612103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:53.612103 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:53.612103 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:53.612103 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:53.612103 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:53.612103 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:53.612103 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:53.612103 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:53.612878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:53.612878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:53.612878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:53.612878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:53.612878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:53.612878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:53.612878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:53.612878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:53.612878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:53.612878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:53.612878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:53.612878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:53.613232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:53.613232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:53.613232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:53.613232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:53.613232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:53.613232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:53.613232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:53.613232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:53.613232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:53.613232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:53.613232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:53.613232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:53.613232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:53.613732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:53.613732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:53.613968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:53.613968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:53.613968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:53.613968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:53.613968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:53.613968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:53.613968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:53.613968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:53.613968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:53.613968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:53.613968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:53.613968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:53.613968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:53.613968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:53.613968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:53.613968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:53.614270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:53.614270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:53.614270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:53.614270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:53.614270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:53.614270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:53.614270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:53.614270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:53.614270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:53.614270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:53.614270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:53.614270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:53.614270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:53.614270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:53.614270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:53.614270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:53.614270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:53.614270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:53.614270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:53.614635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:53.614635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:53.614635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:54.108076 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:54.108359 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:54.108449 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:54.108962 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:54.109144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.109700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:54.109700 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:54.109700 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:54.109700 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:54.109700 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:54.109700 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:54.109700 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.109700 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:54.110414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.110414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:54.110414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:54.110414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:54.110414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.110414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:54.110414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:54.110414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.110414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:54.110414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:54.110414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:54.110414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.110762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:54.110762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:54.110762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:54.110762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:54.110762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:54.110762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:54.110762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:54.110762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:54.110762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:54.110762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:54.110762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:54.110762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:54.110762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:54.111196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.111196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:54.111196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.111196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:54.111196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:54.111196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:54.111196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:54.111196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:54.111196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.111196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:54.111196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:54.111196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.111196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:54.111196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:54.111196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.111683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:54.111683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:54.111683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.111683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:54.111683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:54.111683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.111683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:54.111683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:54.111683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:54.111683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.111683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:54.111683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:54.111683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:54.111683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:54.111683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:54.111683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:54.111683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:54.111683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:54.111683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:54.112288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:54.112288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:54.112288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:54.112288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:54.112288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.112288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:54.119585 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:54.119868 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:54.119980 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:54.120073 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:22:54.120435 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:54.120643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.121129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:54.121129 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:54.121129 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:54.121129 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:54.121129 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:54.121129 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:54.121129 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.121129 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:54.121653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.121653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:54.121653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:54.121653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:54.121653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.121653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:54.121653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:54.121653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:54.121653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.121653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:54.121653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:54.121653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:54.121653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:54.121653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:54.121653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:54.122075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:54.122075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:54.122075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:54.122075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:54.122075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:54.122075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:54.122075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:54.122075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.122075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:54.122356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.122356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:54.122356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:54.122356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:54.122356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.122356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:54.122356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:54.122356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:22:54.122356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:54.122356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:54.122356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.122356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:54.122755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:54.122755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:54.122755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.122755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:54.122755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:54.122755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:54.122755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:54.122755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:54.122755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:54.122755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:54.122755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:54.122755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:54.122755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:54.122755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:54.123157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:54.123157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:54.123157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.123157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:54.123157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.123157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:54.123157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:54.123157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:54.123157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:54.123157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:54.123157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.123157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:54.123157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:54.123157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.123588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:54.123588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:54.123588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.123588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:54.123588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:54.123588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.123588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:54.123588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:54.123588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:22:54.123588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:54.123588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:54.123588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.123588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:54.123588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:54.123588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:54.123588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.123588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:54.123588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:54.123588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:54.124125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:54.124125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:54.124125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:54.124125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:54.124125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:54.124125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:54.124125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:54.124125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:54.124125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:54.124125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:54.124125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.124125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:54.137489 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:54.137822 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:54.137930 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:54.138036 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:54.138429 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:54.138708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.139237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:54.139237 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:54.139237 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:54.139237 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:54.139237 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:54.139237 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:54.139237 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.139237 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:54.139764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.139764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:54.139764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:54.139764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:54.139764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.139764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:54.139764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:54.139764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:54.139764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.139764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:54.139764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:54.139764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:54.139764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:54.139764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:54.139764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:54.140187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:54.140187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:54.140187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:54.140187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:54.140187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:54.140187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:54.140187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:54.140187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.140187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:54.140469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.140469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:54.140469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:54.140469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:54.140469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.140469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:54.140469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:54.140469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:54.140469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:54.140469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.140469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:54.140791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:54.140791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:54.140791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.140791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:54.140791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:54.140791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:54.140791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:54.140791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:54.140791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:54.140791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:54.140791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:54.140791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:54.140791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:54.140791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:54.141198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:54.141198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:54.141198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.141198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:54.141198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.141198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:54.141198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:54.141198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:54.141198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:54.141198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:54.141198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.141198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:54.141198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:54.141198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.141627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:54.141627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:54.141627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.141627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:54.141627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:54.141627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.141627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:54.141627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:54.141627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:54.141627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:54.141627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.141627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:54.141627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:54.141627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:54.141627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.141627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:54.141627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:54.141627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:54.141627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:54.142188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:54.142188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:54.142188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:54.142188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:54.142188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:54.142188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:54.142188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:54.142188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:54.142188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:54.142188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.142188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:54.142986 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:54.143209 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:54.143311 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:54.143432 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:54.143726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:54.143979 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:54.144102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.144561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:54.144561 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:54.144561 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:54.144561 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:54.144561 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:54.144561 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:54.144561 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.144561 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:54.145138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.145138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:54.145138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:54.145138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:54.145138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.145138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:54.145138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:54.145138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.145138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:54.145138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:54.145138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:54.145138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.145500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:54.145500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:54.145500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:54.145500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:54.145500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:54.145500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:54.145500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:54.145500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:54.145500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:54.145500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:54.145500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:54.145500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:54.145500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:54.145877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.145877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:54.145877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.145877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:54.145877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:54.145877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:54.145877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:54.145877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:54.145877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.145877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:54.145877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:54.145877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.145877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:54.145877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:54.145877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.146441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:54.146441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:54.146441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:54.146441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:54.146441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:54.146441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.146441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:54.146441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:54.146441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:54.146441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:54.146441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:54.146441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:54.146441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:54.146441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:54.146441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:54.146441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:54.146441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:54.146441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:54.146441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:54.147102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:54.147102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:54.147102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:54.147102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:54.147102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:54.147102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:58.587006 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:58.587512 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:58.587512 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:58.587864 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:58.612597 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:58.612819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:58.613152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:58.613152 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:58.613152 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:58.613152 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:58.613152 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:58.613152 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:58.613152 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:58.613152 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:58.613763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:58.613763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:58.613763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:58.613763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:58.613763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:58.613763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:58.613763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:58.613763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:58.613763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:58.613763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:58.613763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:58.613763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:58.614006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:58.614006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:58.614006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:58.614006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:58.614006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:58.614006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:58.614006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:58.614006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:58.614006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:58.614006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:58.614006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:58.614006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:58.614006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:58.614207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:58.614207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:58.614509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:58.614509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:58.614509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:58.614509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:58.614509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:58.614509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:58.614509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:58.614509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:58.614509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:58.614509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:58.614509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:58.614509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:58.614509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:58.614509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:58.614509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:58.614509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:58.615339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:22:58.615339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:58.615339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:58.615339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:58.615339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:58.615339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:58.615339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:58.615339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:58.615339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:58.615339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:58.615339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:58.615339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:58.615339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:58.615339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:58.615339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:58.615339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:58.615339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:58.615339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:58.615339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:58.615732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:58.615732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:58.615732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:59.108532 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:59.108787 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:59.108878 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:59.109271 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:59.109415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.109844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:59.109844 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:59.109844 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:59.109844 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:59.109844 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:59.109844 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:59.109844 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.109844 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:59.110471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.110471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:59.110471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:59.110471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:59.110471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.110471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:59.110471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:59.110471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.110471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:59.110471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:59.110471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:59.110471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.110701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:59.110701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:59.110701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:59.110701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:59.110701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:59.110701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:59.110701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:59.110701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:59.110701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:59.110701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:59.110701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:59.110701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:59.110701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:59.110890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.110890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:59.111182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.111182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:59.111182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:59.111182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:59.111182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:59.111182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:59.111182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.111182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:59.111182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:59.111182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.111182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:59.111182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:59.111182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.111182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:59.111182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:59.111182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.111994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:22:59.111994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:59.111994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.111994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:59.111994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:59.111994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:59.111994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.111994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:59.111994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:59.111994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:59.111994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:59.111994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:59.111994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:59.111994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:59.111994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:59.111994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:59.111994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:59.111994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:59.111994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:59.112467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:59.112467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.112467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:59.119992 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:59.120162 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:59.120239 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:59.120349 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:22:59.120596 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:59.120746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.121110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:59.121110 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:59.121110 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:59.121110 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:59.121110 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:59.121110 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:59.121110 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.121110 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:59.121658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.121658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:59.121658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:59.121658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:59.121658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.121658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:59.121658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:59.121658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:59.121658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.121658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:59.121658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:59.121658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:59.121658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:59.121658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:59.121658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:59.121933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:59.121933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:59.121933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:59.121933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:59.121933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:59.121933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:59.121933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:59.121933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.121933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:59.122282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.122282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:59.122282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:59.122282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:59.122282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.122282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:59.122282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:59.122282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:22:59.122282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:59.122282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:59.122282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.122282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:59.122843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:59.122843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:59.122843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.122843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:59.122843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:59.122843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:59.122843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:59.122843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:59.122843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:59.122843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:59.122843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:59.122843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:59.122843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:59.122843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:59.123270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:59.123270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:59.123270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.123270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:59.123270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.123270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:59.123270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:59.123270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:59.123270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:59.123270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:59.123270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.123270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:59.123270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:59.123270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.123496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:59.123496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:59.123496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.123496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:59.123496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:59.123496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.123496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:22:59.123496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:22:59.123496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:22:59.123496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:59.123496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:59.123496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.123496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:59.123496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:59.123496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:59.123496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.123496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:59.123496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:59.123496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:59.123777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:59.123777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:59.123777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:59.123777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:59.123777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:59.123777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:59.123777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:59.123777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:59.123777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:59.123777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:59.123777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.123777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:59.137842 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:59.138002 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:59.138094 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:59.138176 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:59.138460 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:59.138638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.138999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:59.138999 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:59.138999 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:59.138999 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:59.138999 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:59.138999 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:59.138999 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.138999 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:59.139524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.139524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:22:59.139524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:59.139524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:59.139524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.139524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:59.139524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:59.139524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:22:59.139524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.139524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:22:59.139524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:59.139524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:22:59.139524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:59.139524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:22:59.139524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:22:59.139776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:59.139776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:22:59.139776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:59.139776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:22:59.139776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:22:59.139776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:59.139776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:22:59.139776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.139776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:22:59.140142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.140142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:59.140142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:59.140142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:59.140142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.140142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:59.140142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:59.140142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:59.140142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:59.140142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.140142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:59.140361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:59.140361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:59.140361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.140361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:59.140361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:59.140361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:59.140361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:59.140361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:59.140361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:59.140361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:59.140361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:59.140361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:59.140361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:59.140361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:59.140568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:59.140568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:59.140568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.140568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:59.140873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.140873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:59.140873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:59.140873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:59.140873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:59.140873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:59.140873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.140873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:59.140873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:59.140873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.140873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:59.140873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:59.140873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.140873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:59.140873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:59.140873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.140873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:22:59.141353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:22:59.141353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:22:59.141353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:22:59.141353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.141353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:59.141353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:59.141353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:59.141353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.141353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:59.141353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:59.141353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:59.141353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:59.141353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:59.141353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:59.141353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:59.141353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:59.141353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:59.141353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:59.141353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:59.141633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:59.141633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:59.141633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.141633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:22:59.143289 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:22:59.143414 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:22:59.143492 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:22:59.143607 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:22:59.143797 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:59.143971 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:22:59.144047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.144354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:22:59.144354 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:59.144354 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:22:59.144354 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:22:59.144354 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:59.144354 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:22:59.144354 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.144354 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:22:59.144835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.144835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:22:59.144835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:59.144835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:22:59.144835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.144835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:59.144835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:59.144835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.144835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:59.144835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:59.144835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:22:59.144835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.145049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:22:59.145049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:59.145049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:22:59.145049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:59.145049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:22:59.145049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:22:59.145049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:59.145049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:22:59.145049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:59.145049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:22:59.145049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:22:59.145049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:59.145049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:22:59.145233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.145233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:22:59.145505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.145505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:22:59.145505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:22:59.145505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:22:59.145505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:22:59.145505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:22:59.145505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.145505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:22:59.145505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:22:59.145505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.145505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:22:59.145505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:22:59.145505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.145505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:22:59.145505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:22:59.145505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:22:59.146062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:22:59.146062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:22:59.146062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.146062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:59.146062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:22:59.146062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:22:59.146062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:22:59.146062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:22:59.146062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:22:59.146062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:22:59.146062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:22:59.146062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:22:59.146062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:22:59.146062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:22:59.146062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:22:59.146062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:22:59.146062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:22:59.146062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:22:59.146062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:22:59.146360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:22:59.146360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:22:59.146360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:03.587428 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:03.587923 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:03.587923 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:03.588432 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:03.612095 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:03.612281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:03.612641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:03.612641 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:03.612641 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:03.612641 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:03.612641 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:03.612641 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:03.612641 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:03.612641 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:03.613219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:03.613219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:03.613219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:03.613219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:03.613219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:03.613219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:03.613219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:03.613219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:03.613219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:03.613219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:03.613219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:03.613219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:03.613483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:03.613483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:03.613483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:03.613483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:03.613483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:03.613483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:03.613483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:03.613483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:03.613483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:03.613483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:03.613483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:03.613483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:03.613483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:03.613690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:03.613690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:03.614023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:03.614023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:03.614023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:03.614023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:03.614023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:03.614023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:03.614023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:03.614023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:03.614023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:03.614023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:03.614023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:03.614023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:03.614023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:03.614023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:03.614023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:03.614023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:03.614798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:03.614798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:03.614798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:03.614798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:03.614798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:03.614798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:03.614798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:03.614798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:03.614798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:03.614798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:03.614798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:03.614798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:03.614798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:03.614798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:03.614798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:03.614798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:03.614798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:03.614798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:03.614798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:03.615176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:03.615176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:03.615176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:04.108958 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:04.109264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:04.109404 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:04.109844 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:04.109998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.110439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:04.110439 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:04.110439 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:04.110439 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:04.110439 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:04.110439 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:04.110439 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.110439 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:04.111084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.111084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:04.111084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:04.111084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:04.111084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.111084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:04.111084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:04.111084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.111084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:04.111084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:04.111084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:04.111084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.111329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:04.111329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:04.111329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:04.111329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:04.111329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:04.111329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:04.111329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:04.111329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:04.111329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:04.111329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:04.111329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:04.111329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:04.111329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:04.111520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.111520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:04.111844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.111844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:04.111844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:04.111844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:04.111844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:04.111844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:04.111844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.111844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:04.111844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:04.111844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.111844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:04.111844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:04.111844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.111844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:04.111844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:04.111844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.112124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:04.112124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:04.112124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.112124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:04.112124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:04.112124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:04.112124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.112124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:04.112124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:04.112124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:04.112124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:04.112124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:04.112124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:04.112124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:04.112124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:04.112124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:04.112124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:04.112124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:04.112124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:04.112437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:04.112437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.112437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:04.120431 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:04.120666 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:04.120782 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:04.120977 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:23:04.121196 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:04.121373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.121738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:04.121738 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:04.121738 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:04.121738 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:04.121738 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:04.121738 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:04.121738 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.121738 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:04.122288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.122288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:04.122288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:04.122288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:04.122288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.122288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:04.122288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:04.122288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:04.122288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.122288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:04.122288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:04.122288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:04.122288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:04.122288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:04.122288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:04.122669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:04.122669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:04.122669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:04.122669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:04.122669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:04.122669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:04.122669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:04.122669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.122669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:04.122988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.122988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:04.122988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:04.122988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:04.122988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.122988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:04.122988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:04.122988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:23:04.122988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:04.122988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:04.122988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.122988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:04.123544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:04.123544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:04.123544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.123544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:04.123544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:04.123544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:04.123544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:04.123544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:04.123544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:04.123544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:04.123544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:04.123544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:04.123544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:04.123544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:04.123765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:04.123765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:04.123765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.123765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:04.123765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.123765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:04.123765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:04.123765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:04.123765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:04.123765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:04.123765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.123765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:04.123765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:04.123765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.124065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:04.124065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:04.124065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.124065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:04.124065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:04.124065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.124065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:04.124065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:04.124065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:23:04.124065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:04.124065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:04.124065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.124065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:04.124065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:04.124065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:04.124065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.124065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:04.124065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:04.124065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:04.124392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:04.124392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:04.124392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:04.124392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:04.124392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:04.124392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:04.124392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:04.124392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:04.124392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:04.124392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:04.124392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.124392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:04.138492 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:04.138492 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:04.138642 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:04.138642 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:04.139124 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:04.139323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.139762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:04.139762 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:04.139762 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:04.139762 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:04.139762 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:04.139762 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:04.139762 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.139762 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:04.140332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.140332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:04.140332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:04.140332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:04.140332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.140332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:04.140332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:04.140332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:04.140332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.140332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:04.140332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:04.140332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:04.140332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:04.140332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:04.140332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:04.140768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:04.140768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:04.140768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:04.140768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:04.140768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:04.140768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:04.140768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:04.140768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.140768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:04.141046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.141046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:04.141046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:04.141046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:04.141046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.141046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:04.141046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:04.141046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:04.141046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:04.141046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.141046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:04.141342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:04.141342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:04.141342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.141342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:04.141342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:04.141342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:04.141342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:04.141342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:04.141342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:04.141342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:04.141342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:04.141342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:04.141342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:04.141342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:04.141547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:04.141547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:04.141547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.141547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:04.142635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.142635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:04.142635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:04.142635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:04.142635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:04.142635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:04.142635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.142635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:04.142635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:04.142635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.142635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:04.142635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:04.142635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.142635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:04.142635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:04.142635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.142635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:04.143986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:04.143986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:04.143986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:04.143986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.143986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:04.143986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:04.143986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:04.143986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.143986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:04.143986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:04.143986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:04.143986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:04.143986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:04.143986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:04.143986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:04.143986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:04.143986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:04.143986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:04.143986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:04.144409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:04.144409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:04.144409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.144409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:04.144409 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:04.144409 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:04.144409 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:04.144409 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:04.145171 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:04.145171 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:04.145171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.145545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:04.145545 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:04.145545 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:04.145545 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:04.145545 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:04.145545 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:04.145545 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.145545 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:04.146857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.146857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:04.146857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:04.146857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:04.146857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.146857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:04.146857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:04.146857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.146857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:04.146857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:04.146857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:04.146857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.147818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:04.147818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:04.147818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:04.147818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:04.147818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:04.147818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:04.147818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:04.147818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:04.147818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:04.147818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:04.147818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:04.147818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:04.147818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:04.148286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.148286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:04.148286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.148286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:04.148286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:04.148286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:04.148286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:04.148286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:04.148286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.148286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:04.148286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:04.148286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.148286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:04.148286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:04.148286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.149403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:04.149403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:04.149403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:04.149403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:04.149403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:04.149403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.149403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:04.149403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:04.149403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:04.149403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:04.149403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:04.149403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:04.149403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:04.149403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:04.149403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:04.149403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:04.149403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:04.149403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:04.149403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:04.149875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:04.149875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:04.149875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:04.149875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:04.149875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:04.149875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:08.587847 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:08.588486 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:08.588486 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:08.589021 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:08.613781 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:08.614020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:08.614556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:08.614556 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:08.614556 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:08.614556 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:08.614556 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:08.614556 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:08.614556 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:08.614556 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:08.615297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:08.615297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:08.615297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:08.615297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:08.615297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:08.615297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:08.615297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:08.615297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:08.615297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:08.615297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:08.615297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:08.615297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:08.615583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:08.615583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:08.615583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:08.615583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:08.615583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:08.615583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:08.615583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:08.615583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:08.615583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:08.615583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:08.615583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:08.615583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:08.615583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:08.615917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:08.615917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:08.616129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:08.616129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:08.616129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:08.616129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:08.616129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:08.616129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:08.616129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:08.616129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:08.616129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:08.616129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:08.616129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:08.616129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:08.616129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:08.616129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:08.616129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:08.616129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:08.616470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:08.616470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:08.616470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:08.616470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:08.616470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:08.616470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:08.616470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:08.616470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:08.616470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:08.616470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:08.616470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:08.616470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:08.616470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:08.616470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:08.616470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:08.616470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:08.616470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:08.616470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:08.616470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:08.616788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:08.616788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:08.616788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:09.109280 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:09.109805 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:09.109972 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:09.110470 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:09.110648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.111116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:09.111116 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:09.111116 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:09.111116 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:09.111116 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:09.111116 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:09.111116 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.111116 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:09.111864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.111864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:09.111864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:09.111864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:09.111864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.111864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:09.111864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:09.111864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.111864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:09.111864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:09.111864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:09.111864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.112133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:09.112133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:09.112133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:09.112133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:09.112133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:09.112133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:09.112133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:09.112133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:09.112133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:09.112133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:09.112133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:09.112133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:09.112133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:09.112382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.112382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:09.112587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.112587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:09.112587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:09.112587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:09.112587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:09.112587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:09.112587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.112587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:09.112587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:09.112587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.112587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:09.112587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:09.112587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.112587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:09.112587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:09.112587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.112991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:09.112991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:09.112991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.112991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:09.112991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:09.112991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:09.112991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.112991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:09.112991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:09.112991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:09.112991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:09.112991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:09.112991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:09.112991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:09.112991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:09.112991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:09.112991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:09.112991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:09.112991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:09.113381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:09.113381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.113381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:09.120713 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:09.121007 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:09.121105 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:09.121199 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:23:09.121515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:09.121724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.122089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:09.122089 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:09.122089 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:09.122089 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:09.122089 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:09.122089 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:09.122089 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.122089 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:09.122687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.122687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:09.122687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:09.122687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:09.122687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.122687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:09.122687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:09.122687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:09.122687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.122687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:09.122687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:09.122687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:09.122687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:09.122687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:09.122687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:09.123009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:09.123009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:09.123009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:09.123009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:09.123009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:09.123009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:09.123009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:09.123009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.123009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:09.123298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.123298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:09.123298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:09.123298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:09.123298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.123298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:09.123298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:09.123298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:23:09.123298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:09.123298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:09.123298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.123298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:09.123547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:09.123547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:09.123547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.123547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:09.123547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:09.123547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:09.123547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:09.123547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:09.123547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:09.123547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:09.123547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:09.123547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:09.123547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:09.123547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:09.123803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:09.123803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:09.123803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.123803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:09.124037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.124037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:09.124037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:09.124037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:09.124037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:09.124037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:09.124037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.124037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:09.124037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:09.124037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.124037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:09.124037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:09.124037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.124037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:09.124037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:09.124037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.124694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:09.124694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:09.124694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:23:09.124694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:09.124694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:09.124694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.124694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:09.124694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:09.124694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:09.124694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.124694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:09.124694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:09.124694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:09.124694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:09.124694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:09.124694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:09.124694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:09.124694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:09.125264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:09.125264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:09.125264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:09.125264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:09.125264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:09.125264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.125264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:09.138554 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:09.138854 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:09.138997 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:09.139122 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:09.139422 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:09.139624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.140023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:09.140023 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:09.140023 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:09.140023 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:09.140023 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:09.140023 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:09.140023 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.140023 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:09.140617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.140617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:09.140617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:09.140617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:09.140617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.140617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:09.140617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:09.140617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:09.140617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.140617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:09.140617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:09.140617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:09.140617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:09.140617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:09.140617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:09.140897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:09.140897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:09.140897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:09.140897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:09.140897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:09.140897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:09.140897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:09.140897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.140897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:09.141228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.141228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:09.141228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:09.141228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:09.141228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.141228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:09.141228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:09.141228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:09.141228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:09.141228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.141228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:09.141496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:09.141496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:09.141496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.141496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:09.141496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:09.141496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:09.141496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:09.141496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:09.141496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:09.141496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:09.141496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:09.141496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:09.141496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:09.141496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:09.141730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:09.141730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:09.141730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.141730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:09.141983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.141983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:09.141983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:09.141983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:09.141983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:09.141983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:09.141983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.141983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:09.141983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:09.141983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.141983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:09.141983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:09.141983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.141983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:09.141983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:09.141983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.141983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:09.142529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:09.142529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:09.142529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:09.142529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.142529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:09.142529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:09.142529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:09.142529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.142529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:09.142529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:09.142529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:09.142529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:09.142529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:09.142529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:09.142529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:09.142529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:09.142529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:09.142529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:09.142529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:09.142852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:09.142852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:09.142852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.142852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:09.144055 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:09.144210 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:09.144341 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:09.144425 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:09.144668 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:09.144839 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:09.144919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.145298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:09.145298 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:09.145298 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:09.145298 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:09.145298 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:09.145298 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:09.145298 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.145298 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:09.145887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.145887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:09.145887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:09.145887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:09.145887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.145887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:09.145887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:09.145887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.145887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:09.145887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:09.145887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:09.145887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.146131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:09.146131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:09.146131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:09.146131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:09.146131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:09.146131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:09.146131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:09.146131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:09.146131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:09.146131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:09.146131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:09.146131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:09.146131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:09.146394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.146394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:09.146601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.146601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:09.146601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:09.146601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:09.146601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:09.146601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:09.146601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.146601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:09.146601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:09.146601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.146601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:09.146601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:09.146601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.146601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:09.146601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:09.146601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:09.147224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:09.147224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:09.147224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.147224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:09.147224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:09.147224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:09.147224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:09.147224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:09.147224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:09.147224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:09.147224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:09.147224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:09.147224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:09.147224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:09.147224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:09.147224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:09.147224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:09.147224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:09.147224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:09.147580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:09.147580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:09.147580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:13.588318 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:13.588835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:13.588835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:13.589175 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:13.613794 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:13.614434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:13.616466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:13.616466 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:13.616466 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:13.616466 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:13.616466 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:13.616466 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:13.616466 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:13.616466 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:13.617171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:13.617171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:13.617171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:13.617171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:13.617171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:13.617171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:13.617171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:13.617171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:13.617171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:13.617171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:13.617171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:13.617171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:13.617590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:13.617590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:13.617590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:13.617590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:13.617590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:13.617590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:13.617590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:13.617590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:13.617590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:13.617590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:13.617590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:13.617590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:13.617590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:13.617797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:13.617797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:13.617797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:13.617797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:13.617797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:13.617797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:13.617797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:13.617797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:13.617797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:13.617797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:13.617797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:13.617797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:13.617797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:13.617797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:13.617797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:13.618087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:13.618087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:13.618087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:13.618087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:13.618087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:13.618087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:13.618087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:13.618087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:13.618087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:13.618087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:13.618087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:13.618087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:13.618087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:13.618087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:13.618087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:13.618087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:13.618087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:13.618087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:13.618087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:13.618406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:13.618406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:13.618406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:13.618406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:13.618406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:13.618406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:14.109903 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:14.110194 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:14.110334 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:14.111198 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:14.111198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.111378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:14.111378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:14.111378 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:14.111378 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:14.111378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:14.111378 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:14.111378 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.111378 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:14.111909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.111909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:14.111909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:14.111909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:14.111909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.111909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:14.111909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:14.111909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.111909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:14.111909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:14.111909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:14.111909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.112153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:14.112153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:14.112153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:14.112153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:14.112153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:14.112153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:14.112153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:14.112153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:14.112153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:14.112153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:14.112153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:14.112153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:14.112153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:14.112369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.112369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:14.112634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.112634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:14.112634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:14.112634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:14.112634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:14.112634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:14.112634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.112634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:14.112634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:14.112634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.112634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:14.112634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:14.112634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.112634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:14.112634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:14.112634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.113024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:14.113024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:14.113024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.113024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:14.113024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:14.113024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:14.113024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.113024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:14.113024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:14.113024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:14.113024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:14.113024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:14.113024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:14.113024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:14.113024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:14.113024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:14.113024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:14.113024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:14.113024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:14.113327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:14.113327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.113327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:14.121478 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:14.121759 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:14.121863 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:14.121974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:23:14.122709 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:14.122709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.122862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:14.122862 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:14.122862 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:14.122862 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:14.122862 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:14.122862 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:14.122862 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.122862 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:14.123337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.123337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:14.123337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:14.123337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:14.123337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.123337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:14.123337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:14.123337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:14.123337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.123337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:14.123337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:14.123337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:14.123337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:14.123337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:14.123337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:14.123606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:14.123606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:14.123606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:14.123606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:14.123606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:14.123606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:14.123606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:14.123606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.123606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:14.123967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.123967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:14.123967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:14.123967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:14.123967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.123967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:14.123967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:14.123967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:23:14.123967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:14.123967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:14.123967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.123967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:14.124637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:14.124637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:14.124637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.124637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:14.124637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:14.124637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:14.124637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:14.124637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:14.124637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:14.124637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:14.124637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:14.124637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:14.124637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:14.124637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:14.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:14.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:14.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:14.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:14.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:14.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:14.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:14.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:14.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:14.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:14.125034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.125269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:14.125269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:14.125269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.125269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:14.125269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:14.125269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.125269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:14.125269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:14.125269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:23:14.125269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:14.125269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:14.125269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.125269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:14.125269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:14.125269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:14.125269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.125269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:14.125269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:14.125269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:14.125556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:14.125556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:14.125556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:14.125556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:14.125556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:14.125556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:14.125556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:14.125556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:14.125556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:14.125556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:14.125556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.125556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:14.138914 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:14.139228 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:14.139358 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:14.139430 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:14.140310 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:14.140310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.140442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:14.140442 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:14.140442 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:14.140442 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:14.140442 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:14.140442 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:14.140442 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.140442 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:14.140887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.140887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:14.140887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:14.140887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:14.140887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.140887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:14.140887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:14.140887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:14.140887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.140887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:14.140887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:14.140887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:14.140887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:14.140887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:14.140887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:14.141152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:14.141152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:14.141152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:14.141152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:14.141152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:14.141152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:14.141152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:14.141152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.141152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:14.141498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.141498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:14.141498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:14.141498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:14.141498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.141498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:14.141498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:14.141498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:14.141498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:14.141498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.141498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:14.142023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:14.142023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:14.142023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.142023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:14.142023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:14.142023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:14.142023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:14.142023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:14.142023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:14.142023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:14.142023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:14.142023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:14.142023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:14.142023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:14.142410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:14.142410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:14.142410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.142410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:14.142410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.142410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:14.142410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:14.142410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:14.142410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:14.142410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:14.142410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.142410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:14.142410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:14.142410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.142682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:14.142682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:14.142682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.142682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:14.142682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:14.142682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.142682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:14.142682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:14.142682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:14.142682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:14.142682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.142682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:14.142682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:14.142682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:14.142682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.142682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:14.142682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:14.142682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:14.142682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:14.142981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:14.142981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:14.142981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:14.142981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:14.142981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:14.142981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:14.142981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:14.142981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:14.142981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:14.142981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.142981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:14.144357 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:14.144580 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:14.144684 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:14.144794 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:14.145536 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:14.145536 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:14.145536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.145717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:14.145717 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:14.145717 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:14.145717 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:14.145717 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:14.145717 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:14.145717 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.145717 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:14.146136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.146136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:14.146136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:14.146136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:14.146136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.146136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:14.146136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:14.146136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.146136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:14.146136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:14.146136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:14.146136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.146363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:14.146363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:14.146363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:14.146363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:14.146363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:14.146363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:14.146363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:14.146363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:14.146363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:14.146363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:14.146363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:14.146363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:14.146363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:14.146550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.146550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:14.146787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.146787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:14.146787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:14.146787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:14.146787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:14.146787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:14.146787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.146787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:14.146787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:14.146787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.146787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:14.146787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:14.146787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.146787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:14.146787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:14.146787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:14.147539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:14.147539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:14.147539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.147539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:14.147539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:14.147539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:14.147539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:14.147539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:14.147539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:14.147539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:14.147539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:14.147539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:14.147539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:14.147539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:14.147539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:14.147539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:14.147539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:14.147539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:14.147539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:14.147885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:14.147885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:14.147885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:18.588701 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:18.589567 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:18.589694 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:18.590390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:18.624353 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:18.624561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:18.625063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:18.625063 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:18.625063 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:18.625063 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:18.625063 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:18.625063 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:18.625063 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:18.625063 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:18.625731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:18.625731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:18.625731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:18.625731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:18.625731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:18.625731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:18.625731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:18.625731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:18.625731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:18.625731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:18.625731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:18.625731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:18.625987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:18.625987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:18.625987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:18.625987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:18.625987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:18.625987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:18.625987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:18.625987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:18.625987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:18.625987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:18.625987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:18.625987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:18.625987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:18.626305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:18.626305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:18.626460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:18.626460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:18.626460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:18.626460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:18.626460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:18.626460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:18.626460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:18.626460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:18.626460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:18.626460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:18.626460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:18.626460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:18.626460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:18.626460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:18.626460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:18.626460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:18.626852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:18.626852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:18.626852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:18.626852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:18.626852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:18.626852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:18.626852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:18.626852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:18.626852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:18.626852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:18.626852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:18.626852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:18.626852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:18.626852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:18.626852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:18.626852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:18.626852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:18.626852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:18.626852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:18.627200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:18.627200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:18.627200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:19.110322 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:19.110684 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:19.111654 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:19.111990 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:19.112135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.112581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:19.112581 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:19.112581 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:19.112581 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:19.112581 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:19.112581 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:19.112581 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.112581 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:19.113481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.113481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:19.113481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:19.113481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:19.113481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.113481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:19.113481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:19.113481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.113481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:19.113481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:19.113481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:19.113481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.113891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:19.113891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:19.113891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:19.113891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:19.113891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:19.113891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:19.113891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:19.113891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:19.113891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:19.113891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:19.113891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:19.113891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:19.113891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:19.114220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.114220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:19.114503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.114503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:19.114503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:19.114503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:19.114503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:19.114503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:19.114503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.114503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:19.114503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:19.114503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.114503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:19.114503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:19.114503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.114503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:19.114503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:19.114503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.114981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:19.114981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:19.114981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.114981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:19.114981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:19.114981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:19.114981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.114981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:19.114981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:19.114981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:19.114981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:19.114981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:19.114981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:19.114981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:19.114981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:19.114981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:19.114981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:19.114981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:19.114981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:19.115533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:19.115533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.115533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:19.121902 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:19.122226 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:19.122355 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:19.122452 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:23:19.122736 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:19.122885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.123344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:19.123344 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:19.123344 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:19.123344 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:19.123344 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:19.123344 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:19.123344 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.123344 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:19.123830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.123830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:19.123830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:19.123830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:19.123830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.123830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:19.123830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:19.123830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:19.123830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.123830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:19.123830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:19.123830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:19.123830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:19.123830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:19.123830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:19.124130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:19.124130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:19.124130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:19.124130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:19.124130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:19.124130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:19.124130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:19.124130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.124130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:19.124789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.124789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:19.124789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:19.124789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:19.124789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.124789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:19.124789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:19.124789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:23:19.124789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:19.124789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:19.124789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.124789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:19.125096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:19.125096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:19.125096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.125096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:19.125096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:19.125096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:19.125096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:19.125096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:19.125096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:19.125096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:19.125096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:19.125096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:19.125096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:19.125096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:19.125388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:19.125388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:19.125388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.125388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:19.125684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.125684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:19.125684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:19.125684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:19.125684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:19.125684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:19.125684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.125684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:19.125684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:19.125684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.125684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:19.125684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:19.125684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.125684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:19.125684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:19.125684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.126005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:19.126005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:19.126005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:23:19.126005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:19.126005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:19.126005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.126005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:19.126005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:19.126005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:19.126005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.126005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:19.126005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:19.126005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:19.126005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:19.126005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:19.126005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:19.126005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:19.126005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:19.126347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:19.126347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:19.126347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:19.126347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:19.126347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:19.126347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.126347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:19.139323 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:19.139656 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:19.139762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:19.139892 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:19.140218 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:19.140503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.140982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:19.140982 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:19.140982 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:19.140982 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:19.140982 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:19.140982 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:19.140982 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.140982 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:19.141506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.141506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:19.141506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:19.141506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:19.141506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.141506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:19.141506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:19.141506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:19.141506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.141506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:19.141506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:19.141506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:19.141506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:19.141506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:19.141506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:19.141903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:19.141903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:19.141903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:19.141903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:19.141903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:19.141903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:19.141903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:19.141903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.141903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:19.142158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.142158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:19.142158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:19.142158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:19.142158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.142158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:19.142158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:19.142158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:19.142158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:19.142158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.142158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:19.142525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:19.142525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:19.142525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.142525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:19.142525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:19.142525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:19.142525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:19.142525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:19.142525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:19.142525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:19.142525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:19.142525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:19.142525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:19.142525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:19.142921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:19.142921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:19.142921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.142921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:19.142921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.142921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:19.142921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:19.142921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:19.142921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:19.142921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:19.142921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.142921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:19.142921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:19.142921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.143397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:19.143397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:19.143397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.143397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:19.143397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:19.143397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.143397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:19.143397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:19.143397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:19.143397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:19.143397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.143397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:19.143397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:19.143397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:19.143397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.143397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:19.143397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:19.143397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:19.143397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:19.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:19.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:19.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:19.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:19.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:19.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:19.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:19.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:19.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:19.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.143929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:19.144655 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:19.144878 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:19.144979 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:19.145104 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:19.145439 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:19.145684 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:19.145839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.146371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:19.146371 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:19.146371 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:19.146371 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:19.146371 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:19.146371 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:19.146371 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.146371 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:19.146898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.146898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:19.146898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:19.146898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:19.146898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.146898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:19.146898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:19.146898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.146898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:19.146898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:19.146898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:19.146898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.147297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:19.147297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:19.147297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:19.147297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:19.147297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:19.147297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:19.147297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:19.147297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:19.147297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:19.147297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:19.147297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:19.147297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:19.147297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:19.147628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.147628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:19.147628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.147628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:19.147628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:19.147628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:19.147628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:19.147628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:19.147628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.147628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:19.147628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:19.147628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.147628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:19.147628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:19.147628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.148035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:19.148035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:19.148035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:19.148035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:19.148035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:19.148035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.148035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:19.148035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:19.148035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:19.148035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:19.148035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:19.148035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:19.148035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:19.148035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:19.148035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:19.148035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:19.148035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:19.148035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:19.148035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:19.148509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:19.148509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:19.148509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:19.148509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:19.148509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:19.148509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:23.589757 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:23.590212 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:23.590212 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:23.590801 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:23.624482 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:23.624650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:23.625077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:23.625077 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:23.625077 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:23.625077 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:23.625077 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:23.625077 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:23.625077 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:23.625077 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:23.625694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:23.625694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:23.625694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:23.625694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:23.625694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:23.625694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:23.625694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:23.625694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:23.625694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:23.625694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:23.625694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:23.625694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:23.625928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:23.625928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:23.625928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:23.625928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:23.625928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:23.625928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:23.625928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:23.625928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:23.625928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:23.625928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:23.625928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:23.625928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:23.625928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:23.626148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:23.626148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:23.626408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:23.626408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:23.626408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:23.626408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:23.626408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:23.626408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:23.626408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:23.626408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:23.626408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:23.626408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:23.626408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:23.626408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:23.626408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:23.626408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:23.626408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:23.626408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:23.626828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:23.626828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:23.626828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:23.626828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:23.626828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:23.626828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:23.626828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:23.626828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:23.626828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:23.626828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:23.626828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:23.626828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:23.626828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:23.626828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:23.626828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:23.626828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:23.626828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:23.626828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:23.626828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:23.627190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:23.627190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:23.627190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:24.111223 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:24.111628 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:24.111724 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:24.112296 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:24.112418 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:24.112519 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:24.112617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.113063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:24.113063 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:24.113063 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:24.113063 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:24.113063 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:24.113063 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:24.113063 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.113063 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:24.115110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.115110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:24.115110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:24.115110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:24.115110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.115110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:24.115110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:24.115110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.115110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:24.115110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:24.115110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:24.115110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.115362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:24.115362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:24.115362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:24.115362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:24.115362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:24.115362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:24.115362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:24.115362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:24.115362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:24.115362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:24.115362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:24.115362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:24.115362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:24.115603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.115603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:24.115878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.115878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:24.115878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:24.115878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:24.115878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:24.115878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:24.115878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.115878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:24.115878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:24.115878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.115878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:24.115878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:24.115878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.115878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:24.115878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:24.115878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.116393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:24.116393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:24.116393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.116393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:24.116393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:24.116393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:24.116393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.116393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:24.116393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:24.116393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:24.116393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:24.116393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:24.116393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:24.116393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:24.116393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:24.116393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:24.116393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:24.116393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:24.116393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:24.116892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:24.116892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.116892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:24.122268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:24.122482 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:24.122559 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:24.122654 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:23:24.122894 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:24.123056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.123362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:24.123362 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:24.123362 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:24.123362 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:24.123362 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:24.123362 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:24.123362 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.123362 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:24.123844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.123844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:24.123844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:24.123844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:24.123844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.123844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:24.123844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:24.123844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:24.123844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.123844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:24.123844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:24.123844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:24.123844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:24.123844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:24.123844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:24.124328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:24.124328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:24.124328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:24.124328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:24.124328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:24.124328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:24.124328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:24.124328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.124328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:24.124595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.124595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:24.124595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:24.124595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:24.124595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.124595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:24.124595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:24.124595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:23:24.124595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:24.124595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:24.124595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.124595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:24.124957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:24.124957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:24.124957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.124957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:24.124957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:24.124957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:24.124957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:24.124957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:24.124957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:24.124957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:24.124957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:24.124957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:24.124957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:24.124957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:24.125401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:24.125401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:24.125401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.125401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:24.125401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.125401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:24.125401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:24.125401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:24.125401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:24.125401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:24.125401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.125401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:24.125401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:24.125401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:24.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:24.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:24.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:24.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:24.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:24.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:23:24.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:24.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:24.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:24.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:24.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:24.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:24.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:24.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:24.126317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:24.126317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:24.126317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:24.126317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:24.126317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:24.126317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:24.126317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:24.126317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:24.126317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:24.126317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:24.126317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.126317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:24.139711 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:24.139926 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:24.140020 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:24.140117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:24.140327 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:24.140480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.140825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:24.140825 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:24.140825 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:24.140825 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:24.140825 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:24.140825 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:24.140825 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.140825 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:24.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:24.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:24.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:24.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:24.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:24.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:24.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:24.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:24.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:24.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:24.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:24.141290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:24.141729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:24.141729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:24.141729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:24.141729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:24.141729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:24.141729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:24.141729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:24.141729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.141729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:24.142012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.142012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:24.142012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:24.142012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:24.142012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.142012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:24.142012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:24.142012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:24.142012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:24.142012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.142012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:24.142358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:24.142358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:24.142358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.142358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:24.142358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:24.142358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:24.142358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:24.142358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:24.142358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:24.142358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:24.142358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:24.142358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:24.142358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:24.142358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:24.142750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:24.142750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:24.142750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.142750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:24.142750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.142750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:24.142750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:24.142750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:24.142750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:24.142750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:24.142750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.142750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:24.142750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:24.142750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.143158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:24.143158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:24.143158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.143158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:24.143158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:24.143158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.143158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:24.143158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:24.143158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:24.143158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:24.143158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.143158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:24.143158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:24.143158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:24.143158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.143158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:24.143158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:24.143158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:24.143158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:24.143678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:24.143678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:24.143678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:24.143678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:24.143678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:24.143678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:24.143678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:24.143678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:24.143678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:24.143678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.143678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:24.144935 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:24.145117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:24.145194 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:24.145327 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:24.145499 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:24.145674 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:24.145801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.146163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:24.146163 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:24.146163 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:24.146163 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:24.146163 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:24.146163 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:24.146163 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.146163 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:24.146692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.146692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:24.146692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:24.146692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:24.146692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.146692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:24.146692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:24.146692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.146692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:24.146692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:24.146692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:24.146692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.147045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:24.147045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:24.147045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:24.147045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:24.147045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:24.147045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:24.147045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:24.147045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:24.147045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:24.147045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:24.147045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:24.147045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:24.147045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:24.147395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.147395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:24.147395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.147395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:24.147395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:24.147395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:24.147395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:24.147395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:24.147395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.147395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:24.147395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:24.147395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.147395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:24.147395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:24.147395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.147851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:24.147851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:24.147851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:24.147851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:24.147851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:24.147851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.147851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:24.147851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:24.147851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:24.147851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:24.147851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:24.147851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:24.147851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:24.147851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:24.147851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:24.147851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:24.147851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:24.147851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:24.147851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:24.148343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:24.148343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:24.148343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:24.148343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:24.148343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:24.148343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:28.590280 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:28.590809 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:28.590809 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:28.591352 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:28.622914 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:28.622914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:28.622914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:28.622914 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:28.622914 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:28.622914 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:28.622914 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:28.622914 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:28.622914 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:28.622914 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:28.622914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:28.622914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:28.622914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:28.623430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:28.623430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:28.623430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:28.623430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:28.623430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:28.623430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:28.623430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:28.623430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:28.623430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:28.623430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:28.623430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:28.623430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:28.623430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:28.623430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:28.623698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:28.623698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:28.623698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:28.623698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:28.623698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:28.623698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:28.623698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:28.623698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:28.623698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:28.623698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:28.623698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:28.623698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:28.623921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:28.623921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:28.623921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:28.623921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:28.623921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:28.623921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:28.623921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:28.623921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:28.623921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:28.623921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:28.623921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:28.623921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:28.623921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:28.623921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:28.623921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:28.623921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:28.623921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:28.623921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:28.623921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:28.624322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:28.624322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:28.624322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:28.624322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:28.624322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:28.624322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:28.624322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:28.624322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:28.624322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:28.624322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:28.624322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:28.624322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:28.624322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:28.624322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:28.624322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:28.624322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:28.624322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:29.111731 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:29.112093 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:29.112207 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:29.113087 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:29.113287 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:29.113417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:29.113526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-62594d5a-83d3-40a0-a813-2860efcb9c9d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:29.113624 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:29.113728 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:29.113818 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:29.113912 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:29.114041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.114652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:29.114652 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:29.114652 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:29.114652 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:29.114652 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:29.114652 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:29.114652 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.114652 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:29.115442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.115442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:29.115442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:29.115442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:29.115442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.115442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:29.115442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:29.115442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.115442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:29.115442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:29.115442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:29.115442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.115711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:29.115711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:29.115711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:29.115711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:29.115711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:29.115711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:29.115711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:29.115711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:29.115711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:29.115711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:29.115711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:29.115711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:29.115711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:29.116088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.116088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:29.116176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.116176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:29.116176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:29.116176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:29.116176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:29.116176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:29.116176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.116176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:29.116176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:29.116176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.116176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:29.116176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:29.116176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.116176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:29.116176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:29.116176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.116762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:29.116762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:29.116762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.116762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:29.116762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:29.116762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:29.116762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.116762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:29.116762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:29.116762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:29.116762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:29.116762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:29.116762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:29.116762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:29.116762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:29.116762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:29.116762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:29.116762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:29.116762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:29.117511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:29.117511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.117511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:29.122596 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:29.122886 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:29.123005 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:29.123111 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:23:29.123458 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:29.123654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.124076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:29.124076 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:29.124076 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:29.124076 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:29.124076 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:29.124076 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:29.124076 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.124076 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:29.124699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.124699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:29.124699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:29.124699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:29.124699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.124699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:29.124699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:29.124699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:29.124699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.124699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:29.124699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:29.124699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:29.124699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:29.124699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:29.124699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:29.125215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:29.125215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:29.125215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:29.125215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:29.125215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:29.125215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:29.125215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:29.125215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.125215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:29.125510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.125510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:29.125510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:29.125510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:29.125510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.125510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:29.125510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:29.125510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:23:29.125510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:29.125510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:29.125510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.125510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:29.125720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:29.125720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:29.125720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.125720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:29.125720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:29.125720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:29.125720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:29.125720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:29.125720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:29.125720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:29.125720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:29.125720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:29.125720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:29.125720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:29.125929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:29.125929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:29.125929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.125929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:29.126164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.126164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:29.126164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:29.126164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:29.126164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:29.126164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:29.126164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.126164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:29.126164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:29.126164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.126164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:29.126164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:29.126164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.126164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:29.126164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:29.126164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.126981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:29.126981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:29.126981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:23:29.126981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:29.126981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:29.126981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.126981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:29.126981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:29.126981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:29.126981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.126981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:29.126981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:29.126981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:29.126981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:29.126981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:29.126981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:29.126981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:29.126981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:29.127395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:29.127395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:29.127395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:29.127395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:29.127395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:29.127395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.127395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:29.140034 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:29.140449 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:29.140624 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:29.140740 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:29.141103 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:29.141322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.141884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:29.141884 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:29.141884 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:29.141884 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:29.141884 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:29.141884 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:29.141884 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.141884 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:29.142704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.142704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:29.142704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:29.142704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:29.142704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.142704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:29.142704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:29.142704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:29.142704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.142704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:29.142704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:29.142704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:29.142704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:29.142704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:29.142704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:29.142980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:29.142980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:29.142980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:29.142980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:29.142980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:29.142980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:29.142980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:29.142980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.142980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:29.143645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.143645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:29.143645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:29.143645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:29.143645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.143645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:29.143645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:29.143645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:29.143645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:29.143645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.143645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:29.143904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:29.143904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:29.143904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.143904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:29.143904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:29.143904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:29.143904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:29.143904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:29.143904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:29.143904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:29.143904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:29.143904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:29.143904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:29.143904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:29.144190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:29.144190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:29.144190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.144190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:29.144541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.144541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:29.144541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:29.144541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:29.144541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:29.144541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:29.144541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.144541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:29.144541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:29.144541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.144541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:29.144541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:29.144541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.144541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:29.144541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:29.144541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.144541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:29.144828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:29.144828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:29.144828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:29.144828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.144828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:29.144828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:29.144828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:29.144828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.144828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:29.144828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:29.144828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:29.144828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:29.144828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:29.144828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:29.144828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:29.144828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:29.144828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:29.144828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:29.144828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:29.145151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:29.145151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:29.145151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.145151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:29.156007 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:29.157644 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:29.160205 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:29.161100 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:29.161100 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:29.161266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:29.161824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.162080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:29.162080 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:29.162080 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:29.162080 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:29.162080 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:29.162080 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:29.162080 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.162080 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:29.164638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.164638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:29.164638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:29.164638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:29.164638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.164638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:29.164638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:29.164638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.164638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:29.164638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:29.164638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:29.164638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.165753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:29.165753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:29.165753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:29.165753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:29.165753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:29.165753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:29.165753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:29.165753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:29.165753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:29.165753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:29.165753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:29.165753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:29.165753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:29.166215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.166215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:29.168826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.168826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:29.168826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:29.168826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:29.168826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:29.168826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:29.168826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.168826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:29.168826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:29.168826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.168826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:29.168826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:29.168826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.168826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:29.168826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:29.168826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:29.169857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:29.169857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:29.169857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.169857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:29.169857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:29.169857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:29.169857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:29.169857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:29.169857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:29.169857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:29.169857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:29.169857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:29.169857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:29.169857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:29.169857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:29.169857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:29.169857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:29.169857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:29.169857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:29.171540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:29.171540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:29.171540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:33.590574 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:33.591132 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:33.591217 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:33.592232 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:33.633073 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:33.633320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:33.633888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:33.633888 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:33.633888 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:33.633888 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:33.633888 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:33.633888 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:33.633888 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:33.633888 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:33.634846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:33.634846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:33.634846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:33.634846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:33.634846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:33.634846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:33.634846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:33.634846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:33.634846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:33.634846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:33.634846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:33.634846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:33.635094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:33.635094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:33.635094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:33.635094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:33.635094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:33.635094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:33.635094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:33.635094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:33.635094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:33.635094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:33.635094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:33.635094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:33.635094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:33.635325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:33.635325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:33.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:33.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:33.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:33.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:33.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:33.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:33.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:33.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:33.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:33.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:33.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:33.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:33.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:33.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:33.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:33.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:33.636229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:33.636229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:33.636229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:33.636229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:33.636229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:33.636229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:33.636229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:33.636229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:33.636229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:33.636229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:33.636229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:33.636229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:33.636229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:33.636229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:33.636229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:33.636229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:33.636229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:33.636229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:33.636229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:33.636803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:33.636803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:33.636803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:34.112507 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:34.112823 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:34.114097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:34.114097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:34.114193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:34.115124 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:34.115124 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:34.115124 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:34.115124 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:34.115124 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:34.115367 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-62594d5a-83d3-40a0-a813-2860efcb9c9d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:34.115367 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:34.115367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.115638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:34.115638 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:34.115638 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:34.115638 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:34.115638 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:34.115638 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:34.115638 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.115638 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:34.116649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.116649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:34.116649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:34.116649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:34.116649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.116649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:34.116649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:34.116649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.116649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:34.116649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:34.116649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:34.116649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.116904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:34.116904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:34.116904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:34.116904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:34.116904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:34.116904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:34.116904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:34.116904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:34.116904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:34.116904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:34.116904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:34.116904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:34.116904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:34.117114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.117114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:34.117385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.117385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:34.117385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:34.117385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:34.117385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:34.117385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:34.117385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.117385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:34.117385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:34.117385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.117385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:34.117385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:34.117385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.117385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:34.117385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:34.117385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.117681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:34.117681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:34.117681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.117681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:34.117681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:34.117681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:34.117681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.117681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:34.117681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:34.117681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:34.117681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:34.117681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:34.117681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:34.117681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:34.117681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:34.117681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:34.117681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:34.117681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:34.117681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:34.118024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:34.118024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.118024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:34.123176 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:34.123939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:34.123939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:34.123939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:23:34.124169 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:34.124408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.124902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:34.124902 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:34.124902 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:34.124902 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:34.124902 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:34.124902 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:34.124902 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.124902 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:34.125555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.125555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:34.125555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:34.125555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:34.125555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.125555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:34.125555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:34.125555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:34.125555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.125555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:34.125555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:34.125555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:34.125555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:34.125555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:34.125555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:34.125846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:34.125846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:34.125846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:34.125846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:34.125846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:34.125846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:34.125846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:34.125846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.125846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:34.126170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.126170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:34.126170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:34.126170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:34.126170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.126170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:34.126170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:34.126170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:23:34.126170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:34.126170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:34.126170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.126170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:34.126435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:34.126435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:34.126435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.126435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:34.126435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:34.126435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:34.126435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:34.126435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:34.126435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:34.126435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:34.126435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:34.126435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:34.126435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:34.126435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:34.126696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:34.126696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:34.126696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.126696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:34.127238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.127238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:34.127238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:34.127238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:34.127238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:34.127238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:34.127238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.127238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:34.127238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:34.127238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.127238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:34.127238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:34.127238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.127238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:34.127238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:34.127238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.127550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:34.127550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:34.127550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:23:34.127550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:34.127550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:34.127550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.127550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:34.127550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:34.127550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:34.127550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.127550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:34.127550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:34.127550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:34.127550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:34.127550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:34.127550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:34.127550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:34.127550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:34.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:34.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:34.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:34.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:34.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:34.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.127902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:34.140521 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:34.140839 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:34.140945 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:34.141062 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:34.141451 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:34.141702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.142205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:34.142205 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:34.142205 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:34.142205 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:34.142205 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:34.142205 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:34.142205 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.142205 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:34.142823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.142823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:34.142823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:34.142823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:34.142823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.142823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:34.142823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:34.142823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:34.142823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.142823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:34.142823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:34.142823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:34.142823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:34.142823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:34.142823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:34.143218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:34.143218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:34.143218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:34.143218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:34.143218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:34.143218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:34.143218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:34.143218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.143218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:34.143443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.143443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:34.143443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:34.143443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:34.143443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.143443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:34.143443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:34.143443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:34.143443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:34.143443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.143443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:34.143701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:34.143701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:34.143701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.143701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:34.143701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:34.143701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:34.143701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:34.143701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:34.143701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:34.143701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:34.143701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:34.143701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:34.143701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:34.143701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:34.143958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:34.143958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:34.143958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.143958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:34.144167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.144167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:34.144167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:34.144167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:34.144167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:34.144167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:34.144167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.144167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:34.144167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:34.144167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.144167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:34.144167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:34.144167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.144167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:34.144167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:34.144167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.144167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:34.144563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:34.144563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:34.144563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:34.144563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.144563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:34.144563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:34.144563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:34.144563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.144563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:34.144563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:34.144563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:34.144563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:34.144563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:34.144563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:34.144563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:34.144563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:34.144563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:34.144563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:34.144563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:34.144832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:34.144832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:34.144832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.144832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:34.157882 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:34.158314 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:34.158432 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:34.158607 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:34.158957 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:34.159264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:34.159500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.159952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:34.159952 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:34.159952 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:34.159952 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:34.159952 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:34.159952 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:34.159952 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.159952 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:34.160882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.160882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:34.160882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:34.160882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:34.160882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.160882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:34.160882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:34.160882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.160882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:34.160882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:34.160882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:34.160882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.161145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:34.161145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:34.161145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:34.161145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:34.161145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:34.161145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:34.161145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:34.161145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:34.161145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:34.161145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:34.161145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:34.161145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:34.161145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:34.161389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.161389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:34.161859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.161859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:34.161859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:34.161859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:34.161859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:34.161859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:34.161859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.161859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:34.161859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:34.161859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.161859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:34.161859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:34.161859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.161859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:34.161859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:34.161859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:34.162213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:34.162213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:34.162213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.162213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:34.162213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:34.162213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:34.162213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:34.162213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:34.162213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:34.162213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:34.162213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:34.162213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:34.162213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:34.162213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:34.162213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:34.162213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:34.162213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:34.162213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:34.162213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:34.162520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:34.162520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:34.162520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:38.591099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:38.591585 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:38.591681 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:38.592390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:38.626662 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:38.626854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:38.627431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:38.627431 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:38.627431 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:38.627431 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:38.627431 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:38.627431 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:38.627431 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:38.627431 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:38.628265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:38.628265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:38.628265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:38.628265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:38.628265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:38.628265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:38.628265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:38.628265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:38.628265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:38.628265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:38.628265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:38.628265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:38.628476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:38.628476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:38.628476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:38.628476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:38.628476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:38.628476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:38.628476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:38.628476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:38.628476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:38.628476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:38.628476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:38.628476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:38.628476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:38.628706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:38.628706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:38.629221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:38.629221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:38.629221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:38.629221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:38.629221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:38.629221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:38.629221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:38.629221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:38.629221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:38.629221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:38.629221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:38.629221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:38.629221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:38.629221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:38.629221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:38.629221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:38.629503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:38.629503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:38.629503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:38.629503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:38.629503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:38.629503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:38.629503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:38.629503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:38.629503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:38.629503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:38.629503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:38.629503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:38.629503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:38.629503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:38.629503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:38.629503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:38.629503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:38.629503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:38.629503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:38.629846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:38.629846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:38.629846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:39.112937 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:39.113462 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:39.113673 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:39.115091 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:39.115266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:39.115419 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:39.115648 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-62594d5a-83d3-40a0-a813-2860efcb9c9d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:39.115648 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:39.115734 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:39.115900 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:39.115954 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:39.116043 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:39.116141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.116679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:39.116679 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:39.116679 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:39.116679 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:39.116679 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:39.116679 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:39.116679 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.116679 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:39.117548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.117548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:39.117548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:39.117548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:39.117548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.117548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:39.117548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:39.117548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.117548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:39.117548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:39.117548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:39.117548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.117813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:39.117813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:39.117813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:39.117813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:39.117813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:39.117813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:39.117813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:39.117813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:39.117813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:39.117813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:39.117813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:39.117813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:39.117813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:39.118065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.118065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:39.118536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.118536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:39.118536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:39.118536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:39.118536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:39.118536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:39.118536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.118536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:39.118536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:39.118536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.118536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:39.118536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:39.118536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.118536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:39.118536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:39.118536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.118909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:39.118909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:39.118909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.118909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:39.118909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:39.118909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:39.118909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.118909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:39.118909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:39.118909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:39.118909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:39.118909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:39.118909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:39.118909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:39.118909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:39.118909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:39.118909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:39.118909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:39.118909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:39.119368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:39.119368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.119368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:39.123433 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:39.123694 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:39.123798 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:39.123897 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:23:39.124354 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:39.124408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.124812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:39.124812 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:39.124812 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:39.124812 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:39.124812 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:39.124812 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:39.124812 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.124812 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:39.125470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.125470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:39.125470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:39.125470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:39.125470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.125470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:39.125470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:39.125470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:39.125470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.125470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:39.125470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:39.125470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:39.125470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:39.125470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:39.125470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:39.125761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:39.125761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:39.125761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:39.125761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:39.125761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:39.125761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:39.125761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:39.125761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.125761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:39.126371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.126371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:39.126371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:39.126371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:39.126371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.126371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:39.126371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:39.126371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:23:39.126371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:39.126371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:39.126371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.126371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:39.126611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:39.126611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:39.126611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.126611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:39.126611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:39.126611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:39.126611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:39.126611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:39.126611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:39.126611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:39.126611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:39.126611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:39.126611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:39.126611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:39.127058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:39.127058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:39.127058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.127058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:39.127140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.127140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:39.127140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:39.127140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:39.127140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:39.127140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:39.127140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.127140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:39.127140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:39.127140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.127140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:39.127140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:39.127140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.127140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:39.127140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:39.127140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.127402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:39.127402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:39.127402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:23:39.127402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:39.127402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:39.127402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.127402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:39.127402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:39.127402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:39.127402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.127402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:39.127402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:39.127402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:39.127402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:39.127402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:39.127402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:39.127402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:39.127402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:39.127694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:39.127694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:39.127694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:39.127694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:39.127694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:39.127694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.127694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:39.140942 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:39.141766 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:39.141766 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:39.141766 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:39.141903 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:39.142260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.143149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:39.143149 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:39.143149 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:39.143149 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:39.143149 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:39.143149 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:39.143149 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.143149 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:39.143695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.143695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:39.143695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:39.143695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:39.143695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.143695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:39.143695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:39.143695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:39.143695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.143695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:39.143695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:39.143695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:39.143695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:39.143695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:39.143695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:39.144203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:39.144203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:39.144203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:39.144203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:39.144203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:39.144203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:39.144203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:39.144203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.144203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:39.144472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.144472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:39.144472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:39.144472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:39.144472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.144472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:39.144472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:39.144472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:39.144472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:39.144472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.144472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:39.144802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:39.144802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:39.144802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.144802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:39.144802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:39.144802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:39.144802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:39.144802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:39.144802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:39.144802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:39.144802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:39.144802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:39.144802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:39.144802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:39.145201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:39.145201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:39.145201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.145201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:39.145201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.145201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:39.145201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:39.145201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:39.145201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:39.145201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:39.145201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.145201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:39.145201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:39.145201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.145634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:39.145634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:39.145634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.145634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:39.145634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:39.145634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.145634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:39.145634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:39.145634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:39.145634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:39.145634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.145634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:39.145634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:39.145634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:39.145634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.145634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:39.145634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:39.145634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:39.145634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:39.146217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:39.146217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:39.146217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:39.146217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:39.146217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:39.146217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:39.146217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:39.146217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:39.146217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:39.146217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.146217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:39.158363 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:39.158596 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:39.158722 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:39.158906 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:39.159191 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:39.159416 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:39.159660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.160071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:39.160071 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:39.160071 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:39.160071 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:39.160071 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:39.160071 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:39.160071 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.160071 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:39.160746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.160746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:39.160746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:39.160746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:39.160746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.160746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:39.160746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:39.160746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.160746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:39.160746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:39.160746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:39.160746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:39.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:39.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:39.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:39.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:39.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:39.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:39.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:39.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:39.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:39.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:39.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:39.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:39.161305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.161305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:39.162043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.162043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:39.162043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:39.162043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:39.162043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:39.162043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:39.162043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.162043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:39.162043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:39.162043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.162043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:39.162043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:39.162043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.162043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:39.162043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:39.162043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:39.162410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:39.162410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:39.162410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.162410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:39.162410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:39.162410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:39.162410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:39.162410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:39.162410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:39.162410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:39.162410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:39.162410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:39.162410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:39.162410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:39.162410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:39.162410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:39.162410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:39.162410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:39.162410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:39.162741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:39.162741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:39.162741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:43.591620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:43.592178 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:43.592498 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:43.592797 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:43.638582 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:43.638646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:43.639194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:43.639194 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:43.639194 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:43.639194 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:43.639194 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:43.639194 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:43.639194 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:43.639194 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:43.640182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:43.640182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:43.640182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:43.640182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:43.640182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:43.640182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:43.640182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:43.640182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:43.640182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:43.640182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:43.640182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:43.640182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:43.640408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:43.640408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:43.640408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:43.640408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:43.640408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:43.640408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:43.640408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:43.640408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:43.640408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:43.640408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:43.640408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:43.640408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:43.640408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:43.640618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:43.640618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:43.641294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:43.641294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:43.641294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:43.641294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:43.641294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:43.641294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:43.641294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:43.641294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:43.641294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:43.641294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:43.641294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:43.641294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:43.641294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:43.641294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:43.641294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:43.641294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:43.641750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:43.641750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:43.641750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:43.641750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:43.641750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:43.641750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:43.641750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:43.641750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:43.641750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:43.641750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:43.641750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:43.641750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:43.641750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:43.641750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:43.641750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:43.641750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:43.641750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:43.641750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:43.641750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:43.642172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:43.642172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:43.642172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:44.113612 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:44.113732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:44.113822 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:44.115643 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:44.115788 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:44.115893 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:44.116888 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-62594d5a-83d3-40a0-a813-2860efcb9c9d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:44.116888 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:44.116888 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:44.116888 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:44.116888 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:44.117021 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:44.117021 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:44.117021 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-a2536a29-2a84-41cd-b8fb-25208aee2a50, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:44.117021 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:44.117021 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:44.117021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.117402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:44.117402 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:44.117402 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:44.117402 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:44.117402 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:44.117402 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:44.117402 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.117402 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:44.118074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.118074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:44.118074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:44.118074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:44.118074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.118074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:44.118074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:44.118074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.118074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:44.118074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:44.118074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:44.118074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.118570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:44.118570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:44.118570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:44.118570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:44.118570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:44.118570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:44.118570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:44.118570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:44.118570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:44.118570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:44.118570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:44.118570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:44.118570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:44.119575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.119575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:44.119659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.119659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:44.119659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:44.119659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:44.119659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:44.119659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:44.119659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.119659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:44.119659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:44.119659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.119659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:44.119659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:44.119659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.119659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:44.119659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:44.119659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.120237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:44.120237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:44.120237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.120237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:44.120237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:44.120237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:44.120237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.120237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:44.120237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:44.120237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:44.120237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:44.120237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:44.120237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:44.120237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:44.120237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:44.120237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:44.120237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:44.120237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:44.120237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:44.124619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:44.124619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.124619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:44.124619 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:44.124619 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:44.124619 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:44.124619 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:23:44.124619 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:44.124776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.124776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:44.124776 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:44.124776 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:44.124776 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:44.124776 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:44.124776 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:44.124776 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.124776 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:44.125188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.125188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:44.125188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:44.125188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:44.125188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.125188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:44.125188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:44.125188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:44.125188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.125188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:44.125188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:44.125188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:44.125188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:44.125188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:44.125188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:44.125758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:44.125758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:44.125758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:44.125758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:44.125758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:44.125758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:44.125758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:44.125758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.125758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:44.125758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.125758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:44.125758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:44.125758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:44.125758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.126033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:44.126033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:44.126033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:23:44.126033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:44.126033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:44.126033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.126033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:44.126033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:44.126033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:44.126033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.126033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:44.126033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:44.126033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:44.126225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:44.126225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:44.126225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:44.126225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:44.126225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:44.126225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:44.126225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:44.126225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:44.126225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:44.126225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:44.126225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.126225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:44.126432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.126432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:44.126432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:44.126432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:44.126432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:44.126432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:44.126432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.126432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:44.126432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:44.126432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.126432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:44.126432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:44.126432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.126432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:44.126432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:44.126432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.126669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:44.126669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:44.126669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:23:44.126669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:44.126669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:44.126669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.126669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:44.126669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:44.126669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:44.126669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.126669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:44.126669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:44.126669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:44.126669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:44.126669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:44.126669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:44.126669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:44.126669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:44.126955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:44.126955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:44.126955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:44.126955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:44.126955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:44.126955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.126955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:44.141330 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:44.142524 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:44.142524 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:44.142524 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:44.142524 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:44.142524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.142956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:44.142956 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:44.142956 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:44.142956 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:44.142956 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:44.142956 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:44.142956 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.142956 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:44.143325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.143325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:44.143325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:44.143325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:44.143325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.143325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:44.143325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:44.143325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:44.143325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.143325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:44.143325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:44.143325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:44.143325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:44.143325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:44.143325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:44.143770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:44.143770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:44.143770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:44.143770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:44.143770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:44.143770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:44.143770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:44.143770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.143770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:44.144134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.144134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:44.144134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:44.144134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:44.144134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.144134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:44.144134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:44.144134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:44.144134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:44.144134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.144134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:44.144623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:44.144623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:44.144623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.144623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:44.144623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:44.144623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:44.144623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:44.144623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:44.144623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:44.144623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:44.144623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:44.144623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:44.144623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:44.144623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:44.145263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:44.145263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:44.145263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.145263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:44.145263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.145263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:44.145263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:44.145263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:44.145263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:44.145263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:44.145263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.145263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:44.145263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:44.145263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.147148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:44.147148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:44.147148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.147148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:44.147148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:44.147148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.147148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:44.147148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:44.147148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:44.147148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:44.147148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.147148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:44.147148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:44.147148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:44.147148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.147148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:44.147148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:44.147148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:44.147148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:44.147537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:44.147537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:44.147537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:44.147537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:44.147537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:44.147537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:44.147537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:44.147537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:44.147537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:44.147537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.147537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:44.158565 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:44.158834 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:44.158834 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:44.158920 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:44.159201 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:44.159484 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:44.159819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.160934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:44.160934 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:44.160934 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:44.160934 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:44.160934 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:44.160934 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:44.160934 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.160934 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:44.160934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.160934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:44.160934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:44.160934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:44.160934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.160934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:44.160934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:44.161198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.161198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:44.161198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:44.161198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:44.161198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.161198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:44.161198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:44.161198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:44.161198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:44.161198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:44.161198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:44.161198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:44.161198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:44.161198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:44.161463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:44.161463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:44.161463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:44.161463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:44.161463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.161463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:44.161463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.161463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:44.161463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:44.161463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:44.161463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:44.161463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:44.161463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.161463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:44.161682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:44.161981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:44.161981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:44.161981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:44.161981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:44.161981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:44.161981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:44.161981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:44.161981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:44.161981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:44.161981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:48.592138 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:48.592572 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:48.592775 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:48.594231 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:48.636066 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:48.636372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:48.637037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:48.637037 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:48.637037 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:48.637037 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:48.637037 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:48.637037 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:48.637037 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:48.637037 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:48.637764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:48.637764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:48.637764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:48.637764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:48.637764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:48.637764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:48.637764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:48.637764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:48.637764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:48.637764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:48.637764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:48.637764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:48.638023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:48.638023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:48.638023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:48.638023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:48.638023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:48.638023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:48.638023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:48.638023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:48.638023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:48.638023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:48.638023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:48.638023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:48.638023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:48.638216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:48.638216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:48.638435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:48.638435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:48.638435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:48.638435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:48.638435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:48.638435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:48.638435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:48.638435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:48.638435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:48.638435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:48.638435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:48.638435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:48.638435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:48.638435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:48.638435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:48.638435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:48.638705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:48.638705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:48.638705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:48.638705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:48.638705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:48.638705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:48.638705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:48.638705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:48.638705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:48.638705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:48.638705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:48.638705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:48.638705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:48.638705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:48.638705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:48.638705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:48.638705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:48.638705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:48.638705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:48.639004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:48.639004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:48.639004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:49.113857 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:49.114570 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:49.114689 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:49.117807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:49.117807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:49.117807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:49.117807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-62594d5a-83d3-40a0-a813-2860efcb9c9d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:49.117807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:49.118023 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:49.118023 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:49.118023 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:49.118023 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:49.118023 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:49.118141 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:49.118141 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:49.118141 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-a2536a29-2a84-41cd-b8fb-25208aee2a50, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:49.118141 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:49.118141 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:49.118141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.119449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:49.119449 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:49.119449 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:49.119449 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:49.119449 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:49.119449 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:49.119449 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.119449 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:49.120097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.120097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:49.120097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:49.120097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:49.120097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.120097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:49.120097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:49.120097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.120097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:49.120097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:49.120097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:49.120097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.120954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:49.120954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:49.120954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:49.120954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:49.120954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:49.120954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:49.120954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:49.120954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:49.120954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:49.120954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:49.120954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:49.120954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:49.120954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:49.121521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.121521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:49.121521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.121521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:49.121521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:49.121521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:49.121521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:49.121521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:49.121521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.121521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:49.121521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:49.121521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.121521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:49.121521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:49.121521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.122134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:49.122134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:49.122134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.122134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:49.122134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:49.122134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.122134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:49.122134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:49.122134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:49.122134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.122134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:49.122134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:49.122134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:49.122134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:49.122134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:49.122134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:49.122134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:49.122134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:49.122134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:49.122899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:49.122899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:49.122899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:49.122899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:49.122899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.122899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:49.123936 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:49.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:49.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:49.125788 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:23:49.126190 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:49.126869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.128519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:49.128519 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:49.128519 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:49.128519 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:49.128519 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:49.128519 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:49.128519 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.128519 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:49.130174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.130174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:49.130174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:49.130174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:49.130174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.130174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:49.130174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:49.130174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:49.130174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.130174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:49.130174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:49.130174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:49.130174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:49.130174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:49.130174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:49.130919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:49.130919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:49.130919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:49.130919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:49.130919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:49.130919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:49.130919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:49.130919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.130919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:49.130919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.130919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:49.130919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:49.130919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:49.130919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.131369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:49.131369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:49.131369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:23:49.131369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:49.131369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:49.131369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.131369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:49.131369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:49.131369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:49.131369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.131369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:49.131369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:49.131369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:49.131766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:49.131766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:49.131766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:49.131766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:49.131766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:49.131766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:49.131766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:49.131766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:49.131766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:49.131766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:49.131766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.131766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:49.132116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.132116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:49.132116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:49.132116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:49.132116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:49.132116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:49.132116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.132116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:49.132116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:49.132116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.132116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:49.132116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:49.132116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.132116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:49.132116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:49.132116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.132603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:49.132603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:49.132603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:23:49.132603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:49.132603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:49.132603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.132603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:49.132603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:49.132603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:49.132603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.132603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:49.132603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:49.132603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:49.132603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:49.132603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:49.132603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:49.132603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:49.132603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:49.133123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:49.133123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:49.133123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:49.133123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:49.133123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:49.133123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.133123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:49.141672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:49.141964 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:49.142193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:49.142233 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:49.143346 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:49.143745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.144406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:49.144406 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:49.144406 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:49.144406 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:49.144406 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:49.144406 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:49.144406 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.144406 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:49.145325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.145325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:49.145325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:49.145325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:49.145325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.145325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:49.145325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:49.145325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:49.145325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.145325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:49.145325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:49.145325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:49.145325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:49.145325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:49.145325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:49.145957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:49.145957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:49.145957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:49.145957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:49.145957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:49.145957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:49.145957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:49.145957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.145957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:49.145957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.145957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:49.145957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:49.145957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:49.145957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.147159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:49.147159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:49.147159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:49.147159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:49.147159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.147159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:49.147159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:49.147159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:49.147159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.147159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:49.147159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:49.147159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:49.147159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:49.147765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:49.147765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:49.147765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:49.147765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:49.147765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:49.147765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:49.147765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:49.147765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:49.147765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:49.147765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.147765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:49.147765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.148416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:49.148416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:49.148416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:49.148416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:49.148416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:49.148416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.148416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:49.148416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:49.148416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.148416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:49.148416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:49.148416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.148416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:49.148416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:49.148416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.148416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:49.148416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:49.148416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:49.148416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.149435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:49.158972 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:49.159308 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:49.159444 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:49.159642 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:49.162208 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:49.162893 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:49.163414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.164436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:49.164436 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:49.164436 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:49.164436 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:49.164436 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:49.164436 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:49.164436 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.164436 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:49.165196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.165196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:49.165196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:49.165196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:49.165196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.165196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:49.165196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:49.165196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.165196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:49.165196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:49.165196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:49.165196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.166360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:49.166360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:49.166360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:49.166360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:49.166360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:49.166360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:49.166360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:49.166360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:49.166360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:49.166360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:49.166360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:49.166360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:49.166360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:49.166980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.166980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:49.166980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.166980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:49.166980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:49.166980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:49.166980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:49.166980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:49.166980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.166980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:49.166980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:49.166980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.166980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:49.166980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:49.166980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.167314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:49.167314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:49.167314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:49.167314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:49.167314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:49.167314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.167314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:49.167314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:49.167314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:49.167314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:49.167314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:49.167314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:49.167314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:49.167314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:49.167314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:49.167314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:49.167314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:49.167314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:49.167314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:49.167661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:49.167661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:49.167661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:49.167661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:49.167661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:49.167661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:53.592616 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:53.594545 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:53.594545 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:53.594545 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:53.595281 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=240 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:23:53.632092 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:53.632279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:53.634100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:53.634100 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:53.634100 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:53.634100 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:53.634100 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:53.634100 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:53.634100 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:53.634100 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:53.634100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:53.634100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:53.634100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:53.634100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:53.634100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:53.634100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:53.634100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:53.634457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:53.634457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:53.634457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:53.634457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:53.634457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:53.634457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:53.634457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:53.634457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:53.634457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:53.634457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:53.634457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:53.634457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:53.634457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:53.634457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:53.634722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:53.634722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:53.634722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:53.634722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:53.634722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:53.634722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:53.634722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:53.634722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:53.634722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:53.634722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:53.634722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:53.634722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:53.634722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:53.634722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:53.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:53.635352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:53.635352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:53.635352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:53.635352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:53.635352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:53.635352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:53.635352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:53.635352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:53.635352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:54.114674 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:54.115086 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:54.115240 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:54.116592 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.117528 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.117674 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.117846 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-62594d5a-83d3-40a0-a813-2860efcb9c9d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.118216 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.119314 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.119456 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.120039 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.120191 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.122840 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.122840 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-a2536a29-2a84-41cd-b8fb-25208aee2a50, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.123758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.123758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.123758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.123758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.123758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=15): DelStaticRoutesCommand(_result=None, lrouter=neutron-5bcb9456-a2d1-4b5f-9437-1aa4b7c6a444, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.123758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.124141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:54.124141 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:54.124141 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:54.124141 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:54.124141 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:54.124141 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:54.124141 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.124141 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:54.125781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.125781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:54.125781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:54.125781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:54.125781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.125781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:54.125781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:54.125781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.125781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:54.125781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:54.125781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:54.125781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.126373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:54.126373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:54.126373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:54.126373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:54.126373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:54.126373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:54.126373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:54.126373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:54.126373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:54.126373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:54.126373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:54.126373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:54.126373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:54.126966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.126966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:54.129076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.129076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:54.129076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:54.129076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:54.129076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:54.129076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:54.129076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.129076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:54.129076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:54.129076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.129076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:54.129076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:54.129076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.129076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:54.129076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:54.129076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.129906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:54.129906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:54.129906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.129906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:54.129906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:54.129906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:54.129906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.129906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:54.129906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:54.129906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:54.129906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:54.129906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:54.129906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:54.129906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:54.129906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:54.129906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:54.129906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:54.129906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:54.129906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:54.130237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:54.130237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.130237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:54.130237 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:54.130237 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:54.130237 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:54.130237 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:23:54.130237 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.130418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.131112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:54.131112 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:54.131112 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:54.131112 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:54.131112 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:54.131112 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:54.131112 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.131112 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:54.132292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.132292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:54.132292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:54.132292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:54.132292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.132292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:54.132292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:54.132292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:54.132292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.132292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:54.132292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:54.132292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:54.132292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:54.132292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:54.132292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:54.132765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:54.132765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:54.132765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:54.132765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:54.132765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:54.132765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:54.132765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:54.132765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.132765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:54.132765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.132765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:54.132765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:54.132765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:54.132765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.133106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:54.133106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:54.133106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:23:54.133106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:54.133106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:54.133106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.133106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:54.133106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:54.133106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:54.133106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.133106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:54.133106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:54.133106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:54.133382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:54.133382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:54.133382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:54.133382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:54.133382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:54.133382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:54.133382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:54.133382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:54.133382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:54.133382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:54.133382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.133382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:54.135945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.135945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:54.135945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:54.135945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:54.135945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:54.135945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:54.135945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.135945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:54.135945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:54.135945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.135945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:54.135945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:54.135945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.135945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:54.135945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:54.135945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.136647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:54.136647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:54.136647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:23:54.136647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:54.136647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:54.136647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.136647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:54.136647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:54.136647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:54.136647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.136647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:54.136647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:54.136647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:54.136647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:54.136647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:54.136647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:54.136647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:54.136647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:54.137350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:54.137350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:54.137350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:54.137350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:54.137350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:54.137350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.137350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:54.142104 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:54.142410 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:54.142495 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:54.142605 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:54.142851 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.143026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.143373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:54.143373 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:54.143373 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:54.143373 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:54.143373 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:54.143373 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:54.143373 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.143373 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:54.143853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.143853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:54.143853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:54.143853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:54.143853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.143853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:54.143853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:54.143853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:54.143853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.143853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:54.143853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:54.143853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:54.143853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:54.143853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:54.143853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:54.144197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:54.144197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:54.144197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:54.144197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:54.144197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:54.144197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:54.144197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:54.144197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.144197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:54.144439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.144439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:54.144439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:54.144439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:54.144439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.144439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:54.144439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:54.144439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:54.144439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:54.144439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.144439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:54.144631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:54.144631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:54.144631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.144631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:54.144631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:54.144631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:54.144631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:54.144631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:54.144631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:54.144631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:54.144631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:54.144631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:54.144631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:54.144631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:54.144834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:54.144834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:54.144834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.144834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:54.145103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.145103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:54.145103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:54.145103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:54.145103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:54.145103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:54.145103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.145103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:54.145103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:54.145103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.145103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:54.145103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:54.145103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.145103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:54.145103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:54.145103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.145103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:54.145385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:54.145385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:54.145385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:54.145385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.145385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:54.145385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:54.145385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:54.145385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.145385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:54.145385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:54.145385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:54.145385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:54.145385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:54.145385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:54.145385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:54.145385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:54.145385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:54.145385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:54.145385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:54.145678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:54.145678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:54.145678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.145678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:54.159421 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:54.159710 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:54.159806 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:54.159945 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:54.160291 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.160481 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:54.161032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.162792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:54.162792 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:54.162792 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:54.162792 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:54.162792 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:54.162792 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:54.162792 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.162792 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:54.163596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.163596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:54.163596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:54.163596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:54.163596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.163596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:54.163596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:54.163596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.163596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:54.163596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:54.163596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:54.163596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.163813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:54.163813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:54.163813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:54.163813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:54.163813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:54.163813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:54.163813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:54.163813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:54.163813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:54.163813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:54.163813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:54.163813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:54.163813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:54.164020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.164020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:54.164323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.164323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:54.164323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:54.164323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:54.164323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:54.164323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:54.164323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.164323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:54.164323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:54.164323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.164323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:54.164323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:54.164323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.164323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:54.164323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:54.164323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:54.164599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:54.164599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:54.164599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.164599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:54.164599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:54.164599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:54.164599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:54.164599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:54.164599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:54.164599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:54.164599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:54.164599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:54.164599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:54.164599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:54.164599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:54.164599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:54.164599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:54.164599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:54.164599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:54.164886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:54.164886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:54.164886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:58.593501 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:58.593867 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:58.593867 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:58.594433 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:58.647284 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:58.647549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:58.649210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:58.649210 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:58.649210 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:58.649210 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:58.649210 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:58.649210 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:58.649210 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:58.649210 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:58.649210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.054 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:58.649210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:58.649210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:58.649210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:58.649210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:58.649210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:58.649210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:58.649716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:58.649716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:58.649716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:58.649716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:58.649716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:58.649716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:58.649716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:58.649716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:58.649716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:58.649716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:58.649716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:58.649716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:58.649716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:58.649716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:58.649949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:58.649949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:58.649949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:58.649949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:58.649949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:58.649949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:58.649949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:58.649949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:58.649949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:58.649949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:58.649949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:58.649949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:58.649949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:58.649949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:58.650198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:58.650534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:58.650534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:58.650534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:58.650534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:58.650534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:58.650534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:58.650534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:58.650534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:58.650534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:59.115151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:59.115433 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:59.115551 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:59.116836 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.116937 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.117093 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.117163 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.117262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.117388 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.117477 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.117576 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.117718 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.117852 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a2536a29-2a84-41cd-b8fb-25208aee2a50, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.117958 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.118073 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.118156 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.118246 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.118366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-5bcb9456-a2d1-4b5f-9437-1aa4b7c6a444, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.118457 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=15): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.118547 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=16): DelStaticRoutesCommand(_result=None, lrouter=neutron-62594d5a-83d3-40a0-a813-2860efcb9c9d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.118638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.119108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:59.119108 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:59.119108 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:59.119108 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:59.119108 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:59.119108 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:59.119108 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.119108 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:59.119751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.119751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:59.119751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:59.119751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:59.119751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.119751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:59.119751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:59.119751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.119751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:59.119751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:59.119751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:59.119751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.120158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:59.120158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:59.120158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:59.120158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:59.120158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:59.120158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:59.120158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:59.120158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:59.120158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:59.120158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:59.120158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:59.120158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:59.120158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:59.120585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.120585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:59.120585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.120585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:59.120585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:59.120585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:59.120585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:59.120585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:59.120585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.120585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:59.120585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:59.120585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.120585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:59.120585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:59.120585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.121165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:59.121165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:59.121165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.121165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:23:59.121165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:59.121165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.121165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:59.121165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:59.121165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:59.121165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.121165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:59.121165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:59.121165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:59.121165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:59.121165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:59.121165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:59.121165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:59.121165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:59.121165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:59.121770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:59.121770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:59.121770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:59.121770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:59.121770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.121770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:59.129669 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:59.129896 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:59.129994 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:59.130112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:23:59.130690 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.130853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.131286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:59.131286 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:59.131286 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:59.131286 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:59.131286 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:59.131286 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:59.131286 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.131286 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:59.131839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.131839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:59.131839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:59.131839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:59.131839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.131839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:59.131839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:59.131839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:59.131839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.131839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:59.131839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:59.131839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:59.131839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:59.131839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:59.131839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:59.132336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:59.132336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:59.132336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:59.132336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:59.132336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:59.132336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:59.132336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:59.132336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.132336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:59.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:59.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:59.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:59.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:59.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:59.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:23:59.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:59.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:59.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:59.132996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:59.132996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:59.132996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.132996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:59.132996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:59.132996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:59.132996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:59.132996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:59.132996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:59.132996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:59.132996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:59.132996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:59.132996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:59.132996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:59.133460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:59.133460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:59.133460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.133460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:59.133460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.133460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:59.133460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:59.133460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:59.133460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:59.133460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:59.133460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.133460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:59.133460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:59.133460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.133872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:59.133872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:59.133872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.133872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:59.133872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:59.133872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.133872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:23:59.133872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:23:59.133872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:23:59.133872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:59.133872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:59.133872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.133872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:59.133872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:59.133872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:59.133872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.133872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:59.133872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:59.133872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:59.134417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:59.134417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:59.134417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:59.134417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:59.134417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:59.134417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:59.134417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:59.134417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:59.134417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:59.134417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:59.134417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.134417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:59.145725 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:59.146028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:59.146161 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:59.146316 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:59.148569 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.148845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.150311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:59.150311 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:59.150311 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:59.150311 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:59.150311 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:59.150311 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:59.150311 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.150311 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:59.150945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.150945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:23:59.150945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:59.150945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:59.150945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.150945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:59.150945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:59.150945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:23:59.150945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.150945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:23:59.150945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:59.150945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:23:59.150945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:59.150945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:23:59.150945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:23:59.151276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:59.151276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:23:59.151276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:59.151276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:23:59.151276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:23:59.151276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:59.151276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:23:59.151276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.151276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:23:59.151602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.151602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:59.151602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:59.151602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:59.151602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.151602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:59.151602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:59.151602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:59.151602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:59.151602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.151602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:59.152002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:59.152002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:59.152002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.152002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:59.152002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:59.152002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:59.152002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:59.152002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:59.152002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:59.152002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:59.152002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:59.152002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:59.152002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:59.152002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:59.152353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:59.152353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:59.152353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.152353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:59.152437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.152437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:59.152437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:59.152437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:59.152437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:59.152437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:59.152437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.152437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:59.152437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:59.152437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.152437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:59.152437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:59.152437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.152437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:59.152437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:59.152437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.152437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:23:59.152716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:23:59.152716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:23:59.152716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:23:59.152716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.152716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:59.152716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:59.152716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:59.152716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.152716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:59.152716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:59.152716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:59.152716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:59.152716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:59.152716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:59.152716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:59.152716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:59.152716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:59.152716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:59.152716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:59.153492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:59.153492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:59.153492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.153492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:23:59.159748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:23:59.160130 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:23:59.160274 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:23:59.160433 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:23:59.160743 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.160936 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:23:59.161430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.163538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:23:59.163538 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:59.163538 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:23:59.163538 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:23:59.163538 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:59.163538 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:23:59.163538 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.163538 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:23:59.166851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.166851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:23:59.166851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:59.166851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:23:59.166851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.166851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:59.166851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:59.166851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.166851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:59.166851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:59.166851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:23:59.166851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.167211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:23:59.167211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:59.167211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:23:59.167211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:59.167211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:23:59.167211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:23:59.167211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:59.167211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:23:59.167211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:59.167211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:23:59.167211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:23:59.167211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:59.167211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:23:59.167510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.167510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:23:59.168162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.168162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:23:59.168162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:23:59.168162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:23:59.168162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:23:59.168162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:23:59.168162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.168162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:23:59.168162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:23:59.168162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.168162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:23:59.168162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:23:59.168162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.168162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:23:59.168162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:23:59.168162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:23:59.168641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:23:59.168641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:23:59.168641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.168641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:59.168641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:23:59.168641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:23:59.168641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:23:59.168641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:23:59.168641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:23:59.168641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:23:59.168641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:23:59.168641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:23:59.168641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:23:59.168641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:23:59.168641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:23:59.168641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:23:59.168641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:23:59.168641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:23:59.168641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:23:59.169183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:23:59.169183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:23:59.169183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:03.593779 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:03.596352 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:03.596352 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:03.596352 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:03.650779 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:03.650941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:03.651381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:03.651381 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:03.651381 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:03.651381 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:03.651381 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:03.651381 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:03.651381 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:03.651381 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:03.653012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.057 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:03.653012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:03.653012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:03.653012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:03.653012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:03.653012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:03.653012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:03.653012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:03.653012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:03.653012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:03.653012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:03.653012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:03.653286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:03.653286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:03.653286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:03.653286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:03.653286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:03.653286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:03.653286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:03.653286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:03.653286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:03.653286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:03.653286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:03.653286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:03.653286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:03.653550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:03.653550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:03.653550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:03.653550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:03.653550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:03.653550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:03.653550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:03.653550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:03.653550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:03.653550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:03.653550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:03.653550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:03.653550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:03.653550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:03.653550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:03.654064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:03.654064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:03.654064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:03.654064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:03.654064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:03.654064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:03.654064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:03.654064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:03.654064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:03.654064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:03.654064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:03.654064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:03.654064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:03.654064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:03.654064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:03.654064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:03.654064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:03.654064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:03.654064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:03.654432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:03.654432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:03.654432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:03.654432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:03.654432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:03.654432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:04.115533 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:04.117662 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:04.117662 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:04.118166 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:04.118471 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:04.118650 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:04.118745 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:04.118868 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:04.119006 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:04.119141 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:04.119232 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:04.119364 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:04.121060 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:04.121200 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:04.121376 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:04.121548 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:04.121676 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-5bcb9456-a2d1-4b5f-9437-1aa4b7c6a444, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:04.122170 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:04.122170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.123073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:04.123073 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:04.123073 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:04.123073 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:04.123073 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:04.123073 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:04.123073 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.123073 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:04.124452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.124452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:04.124452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:04.124452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:04.124452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.124452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:04.124452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:04.124452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.124452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:04.124452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:04.124452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:04.124452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.124728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:04.124728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:04.124728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:04.124728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:04.124728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:04.124728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:04.124728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:04.124728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:04.124728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:04.124728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:04.124728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:04.124728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:04.124728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:04.124946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.124946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:04.125876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.125876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:04.125876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:04.125876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:04.125876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:04.125876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:04.125876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.125876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:04.125876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:04.125876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.125876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:04.125876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:04.125876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.125876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:04.125876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:04.125876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.126280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:04.126280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:04.126280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.126280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:04.126280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:04.126280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:04.126280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.126280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:04.126280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:04.126280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:04.126280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:04.126280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:04.126280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:04.126280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:04.126280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:04.126280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:04.126280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:04.126280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:04.126280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:04.126959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:04.126959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.126959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:04.130039 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:04.130371 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:04.130705 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:04.130705 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:24:04.131078 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:04.131244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.133443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:04.133443 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:04.133443 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:04.133443 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:04.133443 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:04.133443 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:04.133443 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.133443 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:04.133443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.133443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:04.133443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:04.133443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:04.133443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.133443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:04.133443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:04.133443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:04.133443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.133443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:04.133814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:04.133814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:04.133814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:04.133814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:04.133814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:04.133814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:04.133814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:04.133814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:04.133814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:04.133814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:04.133814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:04.133814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:04.133814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.133814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:04.133814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.134161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:04.134161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:04.134161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:04.134161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.134161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:04.134161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:04.134161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:24:04.134161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:04.134161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:04.134161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.134161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:04.134161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:04.134161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:04.134497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.134497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:04.134497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:04.134497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:04.134497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:04.134497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:04.134497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:04.134497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:04.134497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:04.134497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:04.134497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:04.134497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:04.134497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:04.134497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:04.134738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.134738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:04.135016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.135016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:04.135016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:04.135016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:04.135016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:04.135016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:04.135016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.135016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:04.135016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:04.135016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.135016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:04.135016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:04.135016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.135016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:04.135016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:04.135016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.135418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:04.135418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:04.135418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:24:04.135418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:04.135418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:04.135418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.135418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:04.135418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:04.135418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:04.135418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.135418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:04.135418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:04.135418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:04.135418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:04.135418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:04.135418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:04.135418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:04.135418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:04.135799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:04.135799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:04.135799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:04.135799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:04.135799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:04.135799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.135799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:04.146156 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:04.146704 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:04.146704 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:04.146704 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:04.147240 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:04.147533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.148130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:04.148130 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:04.148130 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:04.148130 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:04.148130 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:04.148130 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:04.148130 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.148130 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:04.150397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.150397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:04.150397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:04.150397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:04.150397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.150397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:04.150397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:04.150397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:04.150397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.150397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:04.150397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:04.150397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:04.150397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:04.150397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:04.150397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:04.150701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:04.150701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:04.150701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:04.150701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:04.150701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:04.150701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:04.150701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:04.150701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.150701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:04.150701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.150701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:04.150701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:04.150701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:04.150701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.150967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:04.150967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:04.150967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:04.150967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:04.150967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.150967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:04.150967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:04.150967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:04.150967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.150967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:04.150967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:04.150967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:04.150967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:04.151332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:04.151332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:04.151332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:04.151332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:04.151332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:04.151332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:04.151332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:04.151332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:04.151332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:04.151332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.151332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:04.151332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.151559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:04.151559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:04.151559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:04.151559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:04.151559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:04.151559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.151559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:04.151559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:04.151559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.151559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:04.151559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:04.151559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.151559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:04.151559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:04.151559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.151559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:04.151559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:04.151559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:04.151559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.151891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:04.160626 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:04.161287 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:04.161287 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:04.161287 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:04.161674 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:04.161958 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:04.162491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.165583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:04.165583 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:04.165583 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:04.165583 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:04.165583 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:04.165583 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:04.165583 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.165583 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:04.166170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.166170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:04.166170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:04.166170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:04.166170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.166170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:04.166170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:04.166170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.166170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:04.166170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:04.166170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:04.166170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.166449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:04.166449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:04.166449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:04.166449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:04.166449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:04.166449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:04.166449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:04.166449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:04.166449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:04.166449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:04.166449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:04.166449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:04.166449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:04.166733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.166733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:04.166854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.166854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:04.166854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:04.166854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:04.166854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:04.166854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:04.166854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.166854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:04.166854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:04.166854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.166854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:04.166854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:04.166854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.166854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:04.166854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:04.166854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:04.167320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:04.167320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:04.167320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.167320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:04.167320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:04.167320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:04.167320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:04.167320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:04.167320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:04.167320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:04.167320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:04.167320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:04.167320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:04.167320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:04.167320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:04.167320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:04.167320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:04.167320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:04.167320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:04.167629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:04.167629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:04.167629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:08.594668 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:08.595351 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:08.595351 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:08.596615 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:08.640584 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:08.641402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:08.642898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:08.642898 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:08.642898 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:08.642898 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:08.642898 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:08.642898 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:08.642898 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:08.642898 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:08.644087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:08.644087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:08.644087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:08.644087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:08.644087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:08.644087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:08.644087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:08.644087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:08.644087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:08.644087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:08.644087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:08.644087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:08.644536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:08.644536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:08.644536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:08.644536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:08.644536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:08.644536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:08.644536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:08.644536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:08.644536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:08.644536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:08.644536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:08.644536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:08.644536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:08.644906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:08.644906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:08.644906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:08.644906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:08.644906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:08.644906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:08.644906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:08.644906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:08.644906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:08.644906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:08.644906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:08.644906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:08.644906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:08.644906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:08.644906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:08.645392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:08.645392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:08.645392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:08.645392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:08.645392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:08.645392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:08.645392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:08.645392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:08.645392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:08.645392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:08.645392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:08.645392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:08.645392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:08.645392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:08.645392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:08.645392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:08.645392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:08.645392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:08.645392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:08.645727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:08.645727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:08.645727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:08.645727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:08.645727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:08.645727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:09.116713 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:09.117267 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:09.117267 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:09.118982 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:09.119855 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:09.120040 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:09.120222 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:09.120453 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:09.120657 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:09.120862 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:09.121620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:09.121877 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:09.122135 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:09.122403 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:09.122812 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:09.123404 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:09.123639 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:09.123826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.124365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:09.124365 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:09.124365 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:09.124365 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:09.124365 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:09.124365 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:09.124365 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.124365 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:09.125182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.125182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:09.125182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:09.125182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:09.125182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.125182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:09.125182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:09.125182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.125182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:09.125182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:09.125182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:09.125182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.125503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:09.125503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:09.125503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:09.125503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:09.125503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:09.125503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:09.125503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:09.125503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:09.125503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:09.125503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:09.125503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:09.125503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:09.125503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:09.125744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.125744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:09.126593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.126593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:09.126593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:09.126593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:09.126593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:09.126593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:09.126593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.126593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:09.126593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:09.126593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.126593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:09.126593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:09.126593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.126593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:09.126593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:09.126593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.126965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:09.126965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:09.126965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.126965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:09.126965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:09.126965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:09.126965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.126965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:09.126965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:09.126965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:09.126965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:09.126965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:09.126965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:09.126965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:09.126965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:09.126965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:09.126965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:09.126965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:09.126965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:09.127458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:09.127458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.127458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:09.130465 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:09.130780 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:09.131081 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:09.131081 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:24:09.134865 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:09.135028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.135632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:09.135632 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:09.135632 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:09.135632 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:09.135632 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:09.135632 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:09.135632 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.135632 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:09.136183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.136183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:09.136183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:09.136183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:09.136183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.136183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:09.136183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:09.136183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:09.136183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.136183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:09.136183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:09.136183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:09.136183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:09.136183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:09.136183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:09.136747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:09.136747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:09.136747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:09.136747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:09.136747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:09.136747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:09.136747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:09.136747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.136747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:09.136996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.136996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:09.136996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:09.136996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:09.136996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.136996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:09.136996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:09.136996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:24:09.136996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:09.136996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:09.136996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.136996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:09.137349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:09.137349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:09.137349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.137349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:09.137349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:09.137349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:09.137349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:09.137349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:09.137349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:09.137349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:09.137349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:09.137349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:09.137349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:09.137349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:09.137703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:09.137703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:09.137703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.137703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:09.137703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.137703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:09.137703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:09.137703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:09.137703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:09.137703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:09.137703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.137703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:09.137703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:09.137703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.138073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:09.138073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:09.138073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.138073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:09.138073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:09.138073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.138073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:09.138073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:09.138073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:24:09.138073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:09.138073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:09.138073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.138073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:09.138073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:09.138073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:09.138073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.138073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:09.138073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:09.138073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:09.138451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:09.138451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:09.138451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:09.138451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:09.138451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:09.138451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:09.138451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:09.138451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:09.138451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:09.138451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:09.138451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.138451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:09.147214 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:09.149451 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:09.149531 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:09.149639 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:09.150063 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:09.150273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.150743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:09.150743 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:09.150743 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:09.150743 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:09.150743 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:09.150743 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:09.150743 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.150743 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:09.151845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.151845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:09.151845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:09.151845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:09.151845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.151845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:09.151845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:09.151845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:09.151845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.151845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:09.151845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:09.151845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:09.151845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:09.151845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:09.151845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:09.152351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:09.152351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:09.152351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:09.152351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:09.152351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:09.152351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:09.152351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:09.152351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.152351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:09.152559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.152559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:09.152559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:09.152559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:09.152559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.152559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:09.152559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:09.152559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:09.152559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:09.152559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.152559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:09.152888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:09.152888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:09.152888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.152888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:09.152888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:09.152888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:09.152888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:09.152888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:09.152888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:09.152888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:09.152888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:09.152888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:09.152888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:09.152888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:09.153335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:09.153335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:09.153335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.153335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:09.153335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.153335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:09.153335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:09.153335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:09.153335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:09.153335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:09.153335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.153335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:09.153335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:09.153335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.153869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:09.153869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:09.153869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.153869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:09.153869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:09.153869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.153869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:09.153869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:09.153869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:09.153869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:09.153869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.153869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:09.153869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:09.153869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:09.153869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.153869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:09.153869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:09.153869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:09.153869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:09.154734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:09.154734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:09.154734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:09.154734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:09.154734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:09.154734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:09.154734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:09.154734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:09.154734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:09.154734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.154734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:09.161330 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:09.161476 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:09.161617 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:09.162157 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:09.163066 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:09.163244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:09.164798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.165389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:09.165389 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:09.165389 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:09.165389 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:09.165389 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:09.165389 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:09.165389 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.165389 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:09.166436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.166436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:09.166436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:09.166436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:09.166436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.166436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:09.166436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:09.166436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.166436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:09.166436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:09.166436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:09.166436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.166831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:09.166831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:09.166831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:09.166831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:09.166831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:09.166831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:09.166831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:09.166831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:09.166831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:09.166831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:09.166831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:09.166831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:09.166831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:09.167232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.167232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:09.170425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.170425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:09.170425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:09.170425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:09.170425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:09.170425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:09.170425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.170425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:09.170425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:09.170425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.170425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:09.170425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:09.170425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.170425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:09.170425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:09.170425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:09.170902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:09.170902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:09.170902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.170902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:09.170902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:09.170902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:09.170902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:09.170902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:09.170902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:09.170902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:09.170902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:09.170902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:09.170902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:09.170902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:09.170902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:09.170902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:09.170902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:09.170902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:09.170902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:09.171512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:09.171512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:09.171512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:13.595616 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:13.595616 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:13.596097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:13.598655 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:13.647293 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:13.649109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:13.649109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:13.649109 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:13.649109 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:13.649109 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:13.649109 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:13.649109 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:13.649109 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:13.649109 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:13.649109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.053 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:13.649109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:13.649109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:13.649109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:13.649109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:13.649109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:13.649607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:13.649607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:13.649607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:13.649607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:13.649607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:13.649607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:13.649607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:13.649607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:13.649607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:13.649607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:13.649607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:13.649607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:13.649607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:13.649607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:13.649977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:13.649977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:13.649977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:13.649977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:13.649977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:13.649977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:13.649977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:13.649977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:13.649977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:13.649977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:13.649977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:13.649977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:13.649977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:13.650379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:13.650914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:13.650914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:13.650914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:13.650914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:13.650914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:13.650914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:13.650914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:13.650914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:13.650914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:13.650914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:13.650914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:13.650914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:14.117446 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:14.118851 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:14.120191 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:14.122605 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:14.122780 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:14.122890 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:14.122998 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:14.123114 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:14.123214 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:14.123332 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:14.123430 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:14.123527 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:14.123624 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:14.123720 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:14.123815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:14.123911 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:14.124005 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:14.124115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.127678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:14.127678 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:14.127678 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:14.127678 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:14.127678 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:14.127678 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:14.127678 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.127678 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:14.129352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.129352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:14.129352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:14.129352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:14.129352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.129352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:14.129352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:14.129352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.129352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:14.129352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:14.129352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:14.129352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.129756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:14.129756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:14.129756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:14.129756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:14.129756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:14.129756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:14.129756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:14.129756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:14.129756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:14.129756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:14.129756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:14.129756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:14.129756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:14.130184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.130184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:14.130608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.130608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:14.130608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:14.130608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:14.130608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:14.130608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:14.130608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.130608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:14.130608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:14.130608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.130608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:14.130608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:14.130608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.130608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:14.130608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:14.130608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.131191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:14.131191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:14.131191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.131191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:14.131191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:14.131191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:14.131191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.131191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:14.131191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:14.131191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:14.131191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:14.131191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:14.131191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:14.131191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:14.131191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:14.131191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:14.131191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:14.131191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:14.131191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:14.131952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:14.131952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.131952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:14.131952 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:14.133296 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:14.133296 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:14.133296 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:24:14.137347 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:14.137580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.139145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:14.139145 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:14.139145 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:14.139145 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:14.139145 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:14.139145 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:14.139145 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.139145 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:14.139891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.139891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:14.139891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:14.139891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:14.139891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.139891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:14.139891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:14.139891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:14.139891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.139891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:14.139891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:14.139891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:14.139891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:14.139891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:14.139891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:14.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:14.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:14.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:14.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:14.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:14.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:14.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:14.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:14.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:14.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:14.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:14.141084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.141639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:14.141639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:14.141639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:24:14.141639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:14.141639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:14.141639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.141639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:14.141639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:14.141639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:14.141639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.141639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:14.141639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:14.141639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:14.142049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:14.142049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:14.142049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:14.142049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:14.142049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:14.142049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:14.142049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:14.142049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:14.142049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:14.142049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:14.142049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.142049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:14.142440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.142440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:14.142440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:14.142440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:14.142440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:14.142440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:14.142440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.142440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:14.142440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:14.142440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.142440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:14.142440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:14.142440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.142440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:14.142440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:14.142440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.142947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:14.142947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:14.142947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:24:14.142947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:14.142947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:14.142947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.142947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:14.142947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:14.142947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:14.142947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.142947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:14.142947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:14.142947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:14.142947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:14.142947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:14.142947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:14.142947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:14.142947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:14.143559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:14.143559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:14.143559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:14.143559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:14.143559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:14.143559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.143559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:14.149545 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:14.150468 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:14.150551 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:14.150656 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:14.151172 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:14.151557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.152270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:14.152270 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:14.152270 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:14.152270 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:14.152270 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:14.152270 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:14.152270 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.152270 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:14.152856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.152856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:14.152856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:14.152856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:14.152856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.152856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:14.152856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:14.152856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:14.152856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.152856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:14.152856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:14.152856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:14.152856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:14.152856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:14.152856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:14.153361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:14.153361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:14.153361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:14.153361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:14.153361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:14.153361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:14.153361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:14.153361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.153361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:14.153698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.153698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:14.153698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:14.153698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:14.153698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.153698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:14.153698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:14.153698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:14.153698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:14.153698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.153698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:14.154083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:14.154083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:14.154083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.154083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:14.154083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:14.154083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:14.154083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:14.154083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:14.154083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:14.154083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:14.154083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:14.154083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:14.154083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:14.154083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:14.154461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:14.154461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:14.154461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.154461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:14.157877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.157877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:14.157877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:14.157877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:14.157877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:14.157877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:14.157877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.157877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:14.157877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:14.157877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.157877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:14.157877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:14.157877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.157877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:14.157877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:14.157877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.157877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:14.158179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:14.158179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:14.158179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:14.158179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.158179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:14.158179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:14.158179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:14.158179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.158179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:14.158179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:14.158179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:14.158179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:14.158179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:14.158179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:14.158179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:14.158179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:14.158179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:14.158179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:14.158179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:14.158476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:14.158476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:14.158476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.158476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:14.161484 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:14.161745 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:14.161825 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:14.161953 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:14.162319 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:14.162586 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:14.162918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.163403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:14.163403 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:14.163403 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:14.163403 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:14.163403 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:14.163403 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:14.163403 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.163403 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:14.164222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.164222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:14.164222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:14.164222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:14.164222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.164222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:14.164222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:14.164222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.164222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:14.164222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:14.164222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:14.164222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.164466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:14.164466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:14.164466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:14.164466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:14.164466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:14.164466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:14.164466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:14.164466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:14.164466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:14.164466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:14.164466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:14.164466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:14.164466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:14.165644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.165644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:14.165644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.165644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:14.165644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:14.165644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:14.165644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:14.165644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:14.165644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.165644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:14.165644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:14.165644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.165644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:14.165644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:14.165644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.165921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:14.165921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:14.165921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:14.165921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:14.165921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:14.165921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.165921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:14.165921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:14.165921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:14.165921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:14.165921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:14.165921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:14.165921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:14.165921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:14.165921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:14.165921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:14.165921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:14.165921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:14.165921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:14.169554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:14.169554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:14.169554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:14.169554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:14.169554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:14.169554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:18.595569 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:18.596038 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:18.596038 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:18.596519 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:18.635738 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:18.635956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:18.636398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:18.636398 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:18.636398 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:18.636398 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:18.636398 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:18.636398 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:18.636398 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:18.636398 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:18.637101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:18.637101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:18.637101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:18.637101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:18.637101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:18.637101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:18.637101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:18.637101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:18.637101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:18.637101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:18.637101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:18.637101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:18.637363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:18.637363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:18.637363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:18.637363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:18.637363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:18.637363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:18.637363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:18.637363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:18.637363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:18.637363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:18.637363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:18.637363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:18.637363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:18.637578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:18.637578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:18.638201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:18.638201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:18.638201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:18.638201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:18.638201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:18.638201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:18.638201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:18.638201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:18.638201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:18.638201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:18.638201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:18.638201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:18.638201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:18.638201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:18.638201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:18.638201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:18.638488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:18.638488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:18.638488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:18.638488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:18.638488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:18.638488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:18.638488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:18.638488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:18.638488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:18.638488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:18.638488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:18.638488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:18.638488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:18.638488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:18.638488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:18.638488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:18.638488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:18.638488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:18.638488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:18.638787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:18.638787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:18.638787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:19.118322 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:19.120531 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:19.120679 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:19.121684 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:19.121859 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:19.124453 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:19.124618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:19.124776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:19.124926 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:19.125077 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:19.125204 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:19.125396 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:19.125581 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:19.125648 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:19.125740 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:19.125853 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:19.125964 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:19.126089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.126667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:19.126667 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:19.126667 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:19.126667 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:19.126667 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:19.126667 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:19.126667 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.126667 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:19.127859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.127859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:19.127859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:19.127859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:19.127859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.127859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:19.127859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:19.127859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.127859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:19.127859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:19.127859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:19.127859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.128314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:19.128314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:19.128314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:19.128314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:19.128314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:19.128314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:19.128314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:19.128314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:19.128314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:19.128314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:19.128314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:19.128314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:19.128314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:19.128727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.128727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:19.129578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.129578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:19.129578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:19.129578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:19.129578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:19.129578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:19.129578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.129578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:19.129578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:19.129578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.129578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:19.129578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:19.129578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.129578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:19.129578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:19.129578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.130138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:19.130138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:19.130138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.130138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:19.130138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:19.130138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:19.130138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.130138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:19.130138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:19.130138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:19.130138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:19.130138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:19.130138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:19.130138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:19.130138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:19.130138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:19.130138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:19.130138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:19.130138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:19.130760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:19.130760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.130760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:19.132315 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:19.132553 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:19.132679 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:19.132806 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:24:19.133075 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:19.133213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.133585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:19.133585 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:19.133585 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:19.133585 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:19.133585 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:19.133585 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:19.133585 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.133585 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:19.134774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.134774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:19.134774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:19.134774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:19.134774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.134774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:19.134774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:19.134774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:19.134774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.134774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:19.134774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:19.134774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:19.134774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:19.134774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:19.134774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:19.135235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:19.135235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:19.135235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:19.135235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:19.135235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:19.135235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:19.135235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:19.135235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.135235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:19.136120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.136120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:19.136120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:19.136120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:19.136120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.136120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:19.136120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:19.136120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:24:19.136120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:19.136120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:19.136120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.136120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:19.136507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:19.136507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:19.136507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.136507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:19.136507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:19.136507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:19.136507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:19.136507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:19.136507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:19.136507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:19.136507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:19.136507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:19.136507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:19.136507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:19.136889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:19.136889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:19.136889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.136889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:19.137697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.137697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:19.137697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:19.137697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:19.137697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:19.137697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:19.137697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.137697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:19.137697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:19.137697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.137697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:19.137697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:19.137697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.137697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:19.137697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:19.137697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.138182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:19.138182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:19.138182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:24:19.138182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:19.138182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:19.138182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.138182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:19.138182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:19.138182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:19.138182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.138182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:19.138182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:19.138182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:19.138182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:19.138182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:19.138182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:19.138182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:19.138182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:19.138763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:19.138763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:19.138763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:19.138763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:19.138763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:19.138763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.138763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:19.150426 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:19.150781 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:19.150912 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:19.151064 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:19.152307 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:19.152564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.153076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:19.153076 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:19.153076 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:19.153076 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:19.153076 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:19.153076 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:19.153076 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.153076 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:19.153642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.153642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:19.153642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:19.153642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:19.153642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.153642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:19.153642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:19.153642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:19.153642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.153642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:19.153642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:19.153642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:19.153642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:19.153642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:19.153642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:19.154031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:19.154031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:19.154031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:19.154031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:19.154031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:19.154031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:19.154031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:19.154031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.154031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:19.154265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.154265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:19.154265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:19.154265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:19.154265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.154265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:19.154265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:19.154265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:19.154265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:19.154265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.154265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:19.154643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:19.154643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:19.154643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.154643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:19.154643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:19.154643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:19.154643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:19.154643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:19.154643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:19.154643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:19.154643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:19.154643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:19.154643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:19.154643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:19.154970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:19.154970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:19.154970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.154970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:19.154970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.154970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:19.154970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:19.154970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:19.154970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:19.154970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:19.154970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.154970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:19.154970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:19.154970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.155492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:19.155492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:19.155492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.155492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:19.155492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:19.155492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.155492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:19.155492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:19.155492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:19.155492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:19.155492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.155492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:19.155492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:19.155492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:19.155492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.155492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:19.155492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:19.155492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:19.155492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:19.155997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:19.155997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:19.155997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:19.155997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:19.155997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:19.155997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:19.155997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:19.155997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:19.155997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:19.155997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.155997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:19.161827 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:19.162122 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:19.162295 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:19.162490 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:19.165540 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:19.165790 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:19.166342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.166898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:19.166898 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:19.166898 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:19.166898 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:19.166898 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:19.166898 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:19.166898 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.166898 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:19.167559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.167559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:19.167559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:19.167559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:19.167559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.167559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:19.167559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:19.167559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.167559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:19.167559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:19.167559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:19.167559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.167782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:19.167782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:19.167782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:19.167782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:19.167782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:19.167782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:19.167782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:19.167782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:19.167782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:19.167782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:19.167782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:19.167782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:19.167782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:19.167972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.167972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:19.168219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.168219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:19.168219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:19.168219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:19.168219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:19.168219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:19.168219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.168219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:19.168219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:19.168219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.168219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:19.168219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:19.168219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.168219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:19.168219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:19.168219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:19.168524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:19.168524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:19.168524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.168524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:19.168524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:19.168524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:19.168524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:19.168524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:19.168524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:19.168524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:19.168524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:19.168524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:19.168524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:19.168524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:19.168524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:19.168524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:19.168524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:19.168524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:19.168524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:19.168840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:19.168840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:19.168840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:23.595920 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:23.596379 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:23.596379 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:23.599338 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:23.657583 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:23.657749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:23.658231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:23.658231 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:23.658231 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:23.658231 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:23.658231 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:23.658231 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:23.658231 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:23.658231 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:23.658982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.062 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:23.658982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:23.658982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:23.658982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:23.658982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:23.658982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:23.658982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:23.658982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:23.658982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:23.658982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:23.658982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:23.658982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:23.659489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:23.659489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:23.659489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:23.659489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:23.659489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:23.659489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:23.659489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:23.659489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:23.659489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:23.659489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:23.659489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:23.659489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:23.659489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:23.659810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:23.659810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:23.659810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:23.659810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:23.659810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:23.659810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:23.659810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:23.659810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:23.659810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:23.659810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:23.659810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:23.659810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:23.659810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:23.659810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:23.659810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:23.660386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:23.660386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:23.660386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:23.660386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:23.660386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:23.660386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:23.660386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:23.660386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:23.660386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:23.660386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:23.660386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:23.660386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:23.660386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:23.660386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:23.660386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:23.660386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:23.660386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:23.660386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:23.660386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:23.660792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:23.660792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:23.660792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:23.660792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:23.660792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:23.660792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:24.118768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:24.119508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:24.119620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:24.120778 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:24.123133 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:24.123949 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:24.123949 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:24.123949 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:24.123949 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:24.124110 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:24.124201 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:24.124373 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:24.124534 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:24.124676 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:24.124814 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:24.124972 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:24.125234 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:24.125401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.129482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:24.129482 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:24.129482 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:24.129482 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:24.129482 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:24.129482 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:24.129482 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.129482 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:24.130247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.130247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:24.130247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:24.130247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:24.130247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.130247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:24.130247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:24.130247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.130247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:24.130247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:24.130247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:24.130247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.130869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:24.130869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:24.130869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:24.130869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:24.130869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:24.130869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:24.130869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:24.130869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:24.130869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:24.130869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:24.130869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:24.130869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:24.130869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:24.132402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.132402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:24.132402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.132402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:24.132402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:24.132402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:24.132402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:24.132402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:24.132402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.132402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:24.132402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:24.132402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.132402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:24.132402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:24.132402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.133134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:24.133134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:24.133134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.133134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:24.133134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:24.133134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.133134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:24.133134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:24.133134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:24.133134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.133134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:24.133134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:24.133134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:24.133134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:24.133134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:24.133134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:24.133134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:24.133134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:24.133134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:24.137770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:24.137770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:24.137770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:24.137770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:24.137770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.137770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:24.137770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:24.137770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:24.137770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:24.137770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:24:24.138079 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:24.138079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.138079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:24.138079 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:24.138079 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:24.138079 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:24.138079 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:24.138079 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:24.138079 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.138079 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:24.138079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.138079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:24.138079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:24.138079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:24.138407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.138407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:24.138407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:24.138407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:24.138407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.138407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:24.138407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:24.138407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:24.138407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:24.138407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:24.138407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:24.138407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:24.138407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:24.138407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:24.138407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:24.138407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:24.138407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:24.138407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:24.138716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.138716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:24.138716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.138716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:24.138716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:24.138716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:24.138716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.138716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:24.138716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:24.138716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:24:24.138716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:24.138716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:24.139102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.139102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:24.139102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:24.139102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:24.139102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.139102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:24.139102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:24.139102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:24.139102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:24.139102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:24.139102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:24.139102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:24.139102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:24.139102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:24.139583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:24.139583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:24.139583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:24.139583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:24.139583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.139583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:24.139583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.139583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:24.139583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:24.139583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:24.139583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:24.139583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:24.139583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.139583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:24.139985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:24.139985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.139985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:24.139985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:24.139985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.139985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:24.139985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:24.139985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.139985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:24.139985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:24.139985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:24:24.139985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:24.139985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:24.139985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.139985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:24.139985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:24.139985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:24.139985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.139985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:24.140935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:24.140935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:24.140935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:24.140935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:24.140935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:24.140935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:24.140935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:24.140935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:24.140935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:24.140935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:24.140935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:24.140935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:24.140935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.140935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:24.150931 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:24.154520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:24.154520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:24.154520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:24.154520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:24.154520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.154692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:24.154692 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:24.154692 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:24.154692 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:24.154692 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:24.154692 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:24.154692 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.154692 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:24.155197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.155197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:24.155197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:24.155197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:24.155197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.155197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:24.155197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:24.155197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:24.155197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.155197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:24.155197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:24.155197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:24.155197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:24.155197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:24.155197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:24.155822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:24.155822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:24.155822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:24.155822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:24.155822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:24.155822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:24.155822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:24.155822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.155822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:24.155822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.155822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:24.155822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:24.155822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:24.155822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.156375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:24.156375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:24.156375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:24.156375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:24.156375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.156375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:24.156375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:24.156375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:24.156375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.156375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:24.156375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:24.156375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:24.156375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:24.156713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:24.156713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:24.156713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:24.156713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:24.156713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:24.156713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:24.156713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:24.156713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:24.156713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:24.156713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.156713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:24.160329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.160329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:24.160329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:24.160329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:24.160329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:24.160329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:24.160329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.160329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:24.160329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:24.160329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.160329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:24.160329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:24.160329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.160329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:24.160329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:24.160329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.160329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:24.160698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:24.160698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:24.160698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:24.160698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.160698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:24.160698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:24.160698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:24.160698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.160698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:24.160698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:24.160698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:24.160698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:24.160698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:24.160698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:24.160698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:24.160698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:24.160698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:24.160698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:24.160698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:24.161037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:24.161037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:24.161037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.161037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:24.162776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:24.163048 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:24.163137 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:24.163284 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:24.163543 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:24.163750 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:24.164105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.164555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:24.164555 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:24.164555 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:24.164555 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:24.164555 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:24.164555 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:24.164555 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.164555 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:24.165119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.165119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:24.165119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:24.165119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:24.165119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.165119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:24.165119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:24.165119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.165119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:24.165119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:24.165119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:24.165119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.165622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:24.165622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:24.165622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:24.165622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:24.165622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:24.165622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:24.165622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:24.165622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:24.165622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:24.165622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:24.165622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:24.165622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:24.165622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:24.166197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.166197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:24.166197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.166197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:24.166197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:24.166197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:24.166197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:24.166197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:24.166197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.166197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:24.166197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:24.166197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.166197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:24.166197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:24.166197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.166944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:24.166944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:24.166944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:24.166944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:24.166944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:24.166944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.166944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:24.166944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:24.166944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:24.166944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:24.166944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:24.166944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:24.166944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:24.166944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:24.166944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:24.166944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:24.166944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:24.166944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:24.166944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:24.167766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:24.167766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:24.167766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:24.167766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:24.167766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:24.167766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:28.596365 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:28.597759 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:28.597883 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:28.598578 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:28.639471 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:28.639655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:28.640191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:28.640191 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:28.640191 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:28.640191 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:28.640191 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:28.640191 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:28.640191 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:28.640191 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:28.640984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:28.640984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:28.640984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:28.640984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:28.640984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:28.640984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:28.640984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:28.640984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:28.640984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:28.640984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:28.640984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:28.640984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:28.641232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:28.641232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:28.641232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:28.641232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:28.641232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:28.641232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:28.641232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:28.641232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:28.641232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:28.641232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:28.641232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:28.641232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:28.641232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:28.641487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:28.641487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:28.642220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:28.642220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:28.642220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:28.642220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:28.642220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:28.642220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:28.642220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:28.642220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:28.642220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:28.642220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:28.642220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:28.642220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:28.642220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:28.642220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:28.642220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:28.642220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:28.642651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:28.642651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:28.642651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:28.642651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:28.642651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:28.642651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:28.642651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:28.642651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:28.642651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:28.642651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:28.642651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:28.642651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:28.642651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:28.642651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:28.642651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:28.642651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:28.642651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:28.642651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:28.642651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:28.643227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:28.643227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:28.643227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:29.119811 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:29.121265 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:29.122353 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:29.124358 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:29.124531 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:29.124657 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:29.124769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:29.124866 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:29.124970 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:29.125078 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:29.125175 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:29.125289 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:29.125387 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:29.125481 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:29.125578 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:29.125674 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:29.125762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:29.125879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.126378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:29.126378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:29.126378 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:29.126378 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:29.126378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:29.126378 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:29.126378 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.126378 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:29.127050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.127050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:29.127050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:29.127050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:29.127050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.127050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:29.127050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:29.127050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.127050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:29.127050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:29.127050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:29.127050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.127344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:29.127344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:29.127344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:29.127344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:29.127344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:29.127344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:29.127344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:29.127344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:29.127344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:29.127344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:29.127344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:29.127344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:29.127344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:29.127671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.127671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:29.127740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.127740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:29.127740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:29.127740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:29.127740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:29.127740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:29.127740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.127740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:29.127740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:29.127740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.127740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:29.127740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:29.127740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.127740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:29.127740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:29.127740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.128068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:29.128068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:29.128068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.128068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:29.128068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:29.128068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:29.128068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.128068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:29.128068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:29.128068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:29.128068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:29.128068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:29.128068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:29.128068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:29.128068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:29.128068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:29.128068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:29.128068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:29.128068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:29.128496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:29.128496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.128496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:29.133107 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:29.133349 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:29.133453 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:29.133555 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:24:29.133895 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:29.134123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.134631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:29.134631 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:29.134631 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:29.134631 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:29.134631 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:29.134631 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:29.134631 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.134631 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:29.135174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.135174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:29.135174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:29.135174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:29.135174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.135174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:29.135174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:29.135174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:29.135174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.135174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:29.135174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:29.135174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:29.135174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:29.135174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:29.135174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:29.135524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:29.135524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:29.135524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:29.135524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:29.135524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:29.135524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:29.135524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:29.135524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.135524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:29.135932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.135932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:29.135932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:29.135932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:29.135932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.135932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:29.135932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:29.135932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:24:29.135932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:29.135932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:29.135932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.135932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:29.136171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:29.136171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:29.136171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.136171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:29.136171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:29.136171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:29.136171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:29.136171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:29.136171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:29.136171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:29.136171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:29.136171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:29.136171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:29.136171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:29.136400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:29.136400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:29.136400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.136400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:29.136903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.136903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:29.136903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:29.136903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:29.136903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:29.136903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:29.136903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.136903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:29.136903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:29.136903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.136903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:29.136903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:29.136903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.136903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:29.136903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:29.136903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.137237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:29.137237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:29.137237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:24:29.137237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:29.137237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:29.137237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.137237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:29.137237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:29.137237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:29.137237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.137237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:29.137237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:29.137237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:29.137237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:29.137237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:29.137237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:29.137237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:29.137237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:29.137551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:29.137551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:29.137551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:29.137551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:29.137551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:29.137551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.137551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:29.153673 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:29.153986 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:29.154146 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:29.154271 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:29.154632 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:29.154829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.155302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:29.155302 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:29.155302 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:29.155302 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:29.155302 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:29.155302 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:29.155302 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.155302 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:29.155843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.155843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:29.155843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:29.155843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:29.155843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.155843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:29.155843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:29.155843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:29.155843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.155843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:29.155843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:29.155843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:29.155843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:29.155843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:29.155843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:29.156327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:29.156327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:29.156327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:29.156327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:29.156327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:29.156327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:29.156327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:29.156327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.156327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:29.156927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.156927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:29.156927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:29.156927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:29.156927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.156927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:29.156927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:29.156927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:29.156927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:29.156927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.156927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:29.157310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:29.157310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:29.157310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.157310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:29.157310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:29.157310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:29.157310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:29.157310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:29.157310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:29.157310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:29.157310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:29.157310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:29.157310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:29.157310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:29.157719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:29.157719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:29.157719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.157719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:29.157719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.157719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:29.157719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:29.157719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:29.157719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:29.157719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:29.157719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.157719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:29.157719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:29.157719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.157998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:29.157998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:29.157998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.157998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:29.157998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:29.157998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.157998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:29.157998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:29.157998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:29.157998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:29.157998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.157998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:29.157998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:29.157998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:29.157998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.157998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:29.157998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:29.157998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:29.157998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:29.158470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:29.158470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:29.158470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:29.158470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:29.158470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:29.158470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:29.158470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:29.158470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:29.158470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:29.158470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.158470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:29.163320 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:29.163556 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:29.163662 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:29.163813 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:29.164124 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:29.164327 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:29.164630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.164981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:29.164981 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:29.164981 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:29.164981 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:29.164981 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:29.164981 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:29.164981 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.164981 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:29.165617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.165617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:29.165617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:29.165617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:29.165617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.165617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:29.165617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:29.165617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.165617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:29.165617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:29.165617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:29.165617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.165932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:29.165932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:29.165932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:29.165932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:29.165932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:29.165932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:29.165932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:29.165932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:29.165932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:29.165932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:29.165932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:29.165932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:29.165932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:29.166164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.166164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:29.166393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.166393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:29.166393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:29.166393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:29.166393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:29.166393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:29.166393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.166393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:29.166393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:29.166393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.166393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:29.166393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:29.166393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.166393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:29.166393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:29.166393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:29.166749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:29.166749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:29.166749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.166749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:29.166749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:29.166749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:29.166749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:29.166749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:29.166749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:29.166749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:29.166749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:29.166749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:29.166749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:29.166749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:29.166749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:29.166749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:29.166749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:29.166749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:29.166749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:29.167063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:29.167063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:29.167063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:33.598041 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:33.598412 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:33.598412 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:33.599571 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:33.636324 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:33.636542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:33.637098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:33.637098 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:33.637098 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:33.637098 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:33.637098 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:33.637098 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:33.637098 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:33.637098 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:33.639125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:33.639125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:33.639125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:33.639125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:33.639125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:33.639125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:33.639125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:33.639125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:33.639125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:33.639125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:33.639125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:33.639125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:33.639374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:33.639374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:33.639374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:33.639374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:33.639374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:33.639374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:33.639374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:33.639374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:33.639374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:33.639374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:33.639374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:33.639374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:33.639374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:33.639585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:33.639585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:33.639954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:33.639954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:33.639954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:33.639954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:33.639954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:33.639954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:33.639954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:33.639954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:33.639954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:33.639954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:33.639954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:33.639954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:33.639954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:33.639954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:33.639954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:33.639954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:33.640836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:33.640836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:33.640836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:33.640836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:33.640836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:33.640836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:33.640836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:33.640836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:33.640836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:33.640836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:33.640836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:33.640836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:33.640836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:33.640836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:33.640836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:33.640836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:33.640836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:33.640836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:33.640836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:33.641361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:33.641361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:33.641361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:34.120432 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:34.120676 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:34.120767 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:34.122529 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:34.122692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:34.122810 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:34.122914 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:34.123074 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:34.123141 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:34.123243 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:34.123354 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:34.123449 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:34.123539 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:34.123633 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:34.123720 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:34.123842 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:34.123911 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:34.124005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.125217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:34.125217 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:34.125217 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:34.125217 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:34.125217 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:34.125217 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:34.125217 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.125217 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:34.125926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.125926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:34.125926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:34.125926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:34.125926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.125926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:34.125926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:34.125926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.125926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:34.125926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:34.125926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:34.125926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.126171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:34.126171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:34.126171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:34.126171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:34.126171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:34.126171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:34.126171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:34.126171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:34.126171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:34.126171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:34.126171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:34.126171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:34.126171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:34.126399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.126399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:34.126878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.126878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:34.126878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:34.126878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:34.126878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:34.126878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:34.126878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.126878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:34.126878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:34.126878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.126878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:34.126878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:34.126878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.126878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:34.126878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:34.126878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.127169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:34.127169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:34.127169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.127169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:34.127169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:34.127169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:34.127169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.127169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:34.127169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:34.127169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:34.127169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:34.127169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:34.127169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:34.127169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:34.127169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:34.127169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:34.127169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:34.127169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:34.127169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:34.127487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:34.127487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.127487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:34.133416 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:34.133751 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:34.133852 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:34.133984 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:24:34.135577 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:34.135753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.136169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:34.136169 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:34.136169 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:34.136169 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:34.136169 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:34.136169 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:34.136169 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.136169 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:34.136737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.136737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:34.136737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:34.136737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:34.136737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.136737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:34.136737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:34.136737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:34.136737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.136737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:34.136737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:34.136737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:34.136737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:34.136737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:34.136737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:34.137044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:34.137044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:34.137044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:34.137044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:34.137044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:34.137044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:34.137044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:34.137044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.137044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:34.137263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.137263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:34.137263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:34.137263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:34.137263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.137263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:34.137263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:34.137263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:24:34.137263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:34.137263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:34.137263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.137263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:34.137471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:34.137471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:34.137471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.137471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:34.137471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:34.137471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:34.137471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:34.137471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:34.137471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:34.137471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:34.137471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:34.137471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:34.137471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:34.137471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:34.137751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:34.137751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:34.137751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.137751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:34.140702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.140702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:34.140702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:34.140702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:34.140702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:34.140702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:34.140702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.140702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:34.140702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:34.140702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.140702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:34.140702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:34.140702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.140702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:34.140702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:34.140702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.141182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:34.141182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:34.141182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:24:34.141182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:34.141182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:34.141182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.141182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:34.141182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:34.141182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:34.141182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.141182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:34.141182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:34.141182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:34.141182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:34.141182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:34.141182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:34.141182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:34.141182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:34.141600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:34.141600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:34.141600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:34.141600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:34.141600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:34.141600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.141600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:34.153981 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:34.154238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:34.154477 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:34.154589 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:34.154906 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:34.155112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.155543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:34.155543 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:34.155543 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:34.155543 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:34.155543 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:34.155543 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:34.155543 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.155543 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:34.156062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.156062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:34.156062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:34.156062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:34.156062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.156062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:34.156062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:34.156062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:34.156062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.156062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:34.156062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:34.156062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:34.156062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:34.156062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:34.156062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:34.156488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:34.156488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:34.156488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:34.156488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:34.156488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:34.156488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:34.156488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:34.156488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.156488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:34.156757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.156757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:34.156757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:34.156757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:34.156757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.156757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:34.156757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:34.156757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:34.156757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:34.156757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.156757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:34.157091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:34.157091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:34.157091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.157091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:34.157091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:34.157091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:34.157091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:34.157091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:34.157091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:34.157091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:34.157091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:34.157091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:34.157091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:34.157091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:34.157498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:34.157498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:34.157498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.157498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:34.157498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.157498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:34.157498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:34.157498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:34.157498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:34.157498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:34.157498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.157498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:34.157498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:34.157498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.158108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:34.158108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:34.158108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.158108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:34.158108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:34.158108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.158108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:34.158108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:34.158108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:34.158108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:34.158108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.158108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:34.158108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:34.158108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:34.158108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.158108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:34.158108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:34.158108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:34.158108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:34.158654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:34.158654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:34.158654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:34.158654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:34.158654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:34.158654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:34.158654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:34.158654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:34.158654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:34.158654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.158654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:34.163604 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:34.164625 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:34.164719 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:34.164894 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:34.165268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:34.165463 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:34.165733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.166173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:34.166173 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:34.166173 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:34.166173 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:34.166173 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:34.166173 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:34.166173 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.166173 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:34.166841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.166841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:34.166841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:34.166841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:34.166841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.166841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:34.166841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:34.166841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.166841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:34.166841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:34.166841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:34.166841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.167165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:34.167165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:34.167165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:34.167165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:34.167165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:34.167165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:34.167165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:34.167165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:34.167165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:34.167165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:34.167165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:34.167165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:34.167165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:34.167469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.167469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:34.167542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.167542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:34.167542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:34.167542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:34.167542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:34.167542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:34.167542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.167542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:34.167542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:34.167542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.167542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:34.167542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:34.167542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.167542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:34.167542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:34.167542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:34.167831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:34.167831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:34.167831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.167831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:34.167831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:34.167831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:34.167831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:34.167831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:34.167831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:34.167831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:34.167831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:34.167831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:34.167831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:34.167831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:34.167831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:34.167831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:34.167831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:34.167831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:34.167831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:34.168152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:34.168152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:34.168152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:38.598459 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:38.599064 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:38.599064 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:38.599523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:38.646312 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:38.646597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:38.647272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:38.647272 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:38.647272 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:38.647272 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:38.647272 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:38.647272 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:38.647272 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:38.647272 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:38.648653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:38.648653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:38.648653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:38.648653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:38.648653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:38.648653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:38.648653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:38.648653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:38.648653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:38.648653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:38.648653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:38.648653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:38.649014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:38.649014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:38.649014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:38.649014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:38.649014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:38.649014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:38.649014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:38.649014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:38.649014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:38.649014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:38.649014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:38.649014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:38.649014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:38.649430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:38.649430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:38.649430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:38.649430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:38.649430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:38.649430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:38.649430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:38.649430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:38.649430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:38.649430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:38.649430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:38.649430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:38.649430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:38.649430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:38.649430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:38.649980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:38.649980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:38.649980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:38.649980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:38.649980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:38.649980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:38.649980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:38.649980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:38.649980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:38.649980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:38.649980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:38.649980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:38.649980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:38.649980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:38.649980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:38.649980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:38.649980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:38.649980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:38.649980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:38.651763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:38.651763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:38.651763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:38.651763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:38.651763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:38.651763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:39.121266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:39.121603 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:39.121741 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:39.125908 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:39.126152 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:39.126350 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:39.126489 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:39.126643 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:39.126794 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:39.126972 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:39.127967 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:39.127967 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:39.127967 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:39.127967 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:39.127967 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:39.128237 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:39.128237 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:39.128237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.129092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:39.129092 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:39.129092 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:39.129092 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:39.129092 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:39.129092 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:39.129092 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.129092 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:39.130157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.130157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:39.130157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:39.130157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:39.130157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.130157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:39.130157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:39.130157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.130157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:39.130157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:39.130157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:39.130157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.130481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:39.130481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:39.130481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:39.130481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:39.130481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:39.130481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:39.130481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:39.130481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:39.130481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:39.130481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:39.130481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:39.130481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:39.130481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:39.130838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.130838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:39.134738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.134738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:39.134738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:39.134738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:39.134738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:39.134738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:39.134738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.134738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:39.134738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:39.134738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.134738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:39.134738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:39.134738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.134738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:39.134738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:39.134738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.135209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:39.135209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:39.135209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.135209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:39.135209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:39.135209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:39.135209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.135209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:39.135209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:39.135209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:39.135209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:39.135209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:39.135209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:39.135209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:39.135209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:39.135209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:39.135209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:39.135209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:39.135209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:39.135660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:39.135660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.135660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:39.135660 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:39.135660 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:39.135660 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:39.135660 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:24:39.135910 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:39.137642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.137642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:39.137642 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:39.137642 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:39.137642 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:39.137642 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:39.137642 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:39.137642 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.137642 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:39.137642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.137642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:39.137642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:39.137642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:39.137642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.137642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:39.137642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:39.137642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:39.138196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.138196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:39.138196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:39.138196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:39.138196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:39.138196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:39.138196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:39.138196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:39.138196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:39.138196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:39.138196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:39.138196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:39.138196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:39.138196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:39.138196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.138196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:39.138746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.138746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:39.138746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:39.138746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:39.138746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.138746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:39.138746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:39.138746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:24:39.138746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:39.138746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:39.138746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.138746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:39.139137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:39.139137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:39.139137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.139137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:39.139137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:39.139137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:39.139137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:39.139137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:39.139137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:39.139137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:39.139137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:39.139137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:39.139137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:39.139137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:39.139645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:39.139645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:39.139645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.139645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:39.139645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.139645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:39.139645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:39.139645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:39.139645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:39.139645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:39.139645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.139645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:39.139645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:39.139645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.140059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:39.140059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:39.140059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.140059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:39.140059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:39.140059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.140059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:39.140059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:39.140059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:24:39.140059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:39.140059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:39.140059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.140059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:39.140059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:39.140059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:39.140059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.140059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:39.140059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:39.140059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:39.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:39.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:39.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:39.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:39.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:39.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:39.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:39.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:39.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:39.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:39.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.140707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:39.155278 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:39.155278 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:39.155278 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:39.155278 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:39.155467 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:39.155614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.156811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:39.156811 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:39.156811 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:39.156811 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:39.156811 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:39.156811 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:39.156811 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.156811 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:39.159135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.159135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:39.159135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:39.159135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:39.159135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.159135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:39.159135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:39.159135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:39.159135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.159135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:39.159135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:39.159135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:39.159135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:39.159135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:39.159135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:39.159494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:39.159494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:39.159494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:39.159494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:39.159494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:39.159494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:39.159494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:39.159494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.159494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:39.159494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.159494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:39.159494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:39.159494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:39.159494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.159754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:39.159754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:39.159754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:39.159754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:39.159754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.159754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:39.159754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:39.159754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:39.159754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.159754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:39.159754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:39.159754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:39.159754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:39.160007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:39.160007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:39.160007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:39.160007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:39.160007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:39.160007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:39.160007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:39.160007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:39.160007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:39.160007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.160007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:39.160007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.160261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:39.160261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:39.160261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:39.160261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:39.160261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:39.160261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.160261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:39.160261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:39.160261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.160261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:39.160261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:39.160261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.160261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:39.160261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:39.160261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.160261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:39.160261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:39.160261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:39.160261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.160608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:39.164596 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:39.164824 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:39.164919 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:39.165068 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:39.165840 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:39.166093 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:39.166514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.166960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:39.166960 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:39.166960 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:39.166960 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:39.166960 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:39.166960 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:39.166960 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.166960 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:39.167575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.167575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:39.167575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:39.167575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:39.167575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.167575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:39.167575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:39.167575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.167575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:39.167575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:39.167575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:39.167575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.167930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:39.167930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:39.167930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:39.167930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:39.167930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:39.167930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:39.167930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:39.167930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:39.167930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:39.167930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:39.167930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:39.167930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:39.167930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:39.168239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.168239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:39.168239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.168239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:39.168239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:39.168239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:39.168239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:39.168239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:39.168239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.168239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:39.168239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:39.168239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.168239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:39.168239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:39.168239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.168548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:39.168548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:39.168548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:39.168548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:39.168548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:39.168548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.168548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:39.168548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:39.168548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:39.168548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:39.168548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:39.168548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:39.168548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:39.168548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:39.168548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:39.168548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:39.168548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:39.168548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:39.168548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:39.168936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:39.168936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:39.168936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:39.168936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:39.168936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:39.168936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:43.599229 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:43.602766 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:43.602766 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:43.602766 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:43.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:43.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:43.696871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:43.696871 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:43.696871 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:43.696871 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:43.696871 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:43.696871 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:43.696871 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:43.696871 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:43.698149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.095 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:43.698149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:43.698149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:43.698149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:43.698149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:43.698149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:43.698149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:43.698149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:43.698149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:43.698149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:43.698149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:43.698149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:43.698603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:43.698603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:43.698603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:43.698603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:43.698603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:43.698603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:43.698603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:43.698603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:43.698603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:43.698603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:43.698603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:43.698603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:43.698603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:43.699005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:43.699005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:43.700058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:43.700058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:43.700058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:43.700058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:43.700058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:43.700058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:43.700058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:43.700058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:43.700058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:43.700058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:43.700058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:43.700058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:43.700058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:43.700058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:43.700058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:43.700058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:43.700682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:43.700682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:43.700682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:43.700682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:43.700682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:43.700682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:43.700682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:43.700682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:43.700682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:43.700682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:43.700682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:43.700682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:43.700682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:43.700682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:43.700682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:43.700682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:43.700682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:43.700682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:43.700682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:43.701312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:43.701312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:43.701312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:44.123514 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:44.123647 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:44.123695 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:44.134874 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:44.134874 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:44.134874 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:44.135033 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:44.135091 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:44.135233 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:44.135297 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:44.135406 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:44.135499 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:44.135596 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:44.135693 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:44.135787 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:44.135889 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:44.136670 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:44.136670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.136670 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:44.137448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:44.137448 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:44.137448 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:44.137448 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:44.137448 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:44.137448 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:44.137448 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.137448 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:44.138378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.014 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.138378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:44.138378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:44.138378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:44.138378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.138378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:44.138378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:44.138378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.138378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:44.138378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:44.138378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:44.138378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.138644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:44.138644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:44.138644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:44.138644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:44.138644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:44.138644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:44.138644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:44.138644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:44.138644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:44.138644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:44.138644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:44.138644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:44.138644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:44.139502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.139502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:44.140896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.140896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:44.140896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:44.140896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:44.140896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:44.140896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:44.140896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.140896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:44.140896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:44.140896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.140896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:44.140896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:44.140896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.140896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:44.140896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:44.140896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.141225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:44.141225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:44.141225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.141225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:44.141225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:44.141225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:44.141225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.141225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:44.141225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:44.141225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:44.141225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:44.141225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:44.141225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:44.141225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:44.141225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:44.141225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:44.141225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:44.141225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:44.141225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:44.141560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:44.141560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.141560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:44.143008 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:44.143008 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:44.143008 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:24:44.145707 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:44.145779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.146937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:44.146937 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:44.146937 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:44.146937 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:44.146937 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:44.146937 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:44.146937 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.146937 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:44.149441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.149441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:44.149441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:44.149441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:44.149441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.149441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:44.149441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:44.149441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:44.149441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.149441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:44.149441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:44.149441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:44.149441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:44.149441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:44.149441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:44.149767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:44.149767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:44.149767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:44.149767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:44.149767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:44.149767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:44.149767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:44.149767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.149767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:44.150426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.150426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:44.150426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:44.150426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:44.150426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.150426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:44.150426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:44.150426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:24:44.150426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:44.150426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:44.150426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.150426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:44.150711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:44.150711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:44.150711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.150711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:44.150711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:44.150711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:44.150711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:44.150711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:44.150711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:44.150711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:44.150711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:44.150711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:44.150711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:44.150711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:44.150941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:44.150941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:44.150941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.150941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:44.153700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.153700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:44.153700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:44.153700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:44.153700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:44.153700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:44.153700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.153700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:44.153700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:44.153700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.153700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:44.153700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:44.153700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.153700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:44.153700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:44.153700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.154008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:44.154008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:44.154008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:24:44.154008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:44.154008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:44.154008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.154008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:44.154008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:44.154008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:44.154008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.154008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:44.154008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:44.154008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:44.154008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:44.154008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:44.154008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:44.154008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:44.154008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:44.154379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:44.154379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:44.154379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:44.154379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:44.154379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:44.154379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.154379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:44.155501 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:44.156034 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:44.156230 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:44.156445 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:44.156714 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:44.156896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.157914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:44.157914 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:44.157914 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:44.157914 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:44.157914 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:44.157914 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:44.157914 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.157914 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:44.160327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.160327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:44.160327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:44.160327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:44.160327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.160327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:44.160327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:44.160327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:44.160327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.160327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:44.160327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:44.160327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:44.160327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:44.160327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:44.160327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:44.160667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:44.160667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:44.160667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:44.160667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:44.160667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:44.160667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:44.160667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:44.160667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.160667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:44.160667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.160667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:44.160667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:44.160667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:44.160667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.160935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:44.160935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:44.160935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:44.160935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:44.160935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.160935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:44.160935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:44.160935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:44.160935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.160935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:44.160935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:44.160935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:44.160935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:44.161182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:44.161182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:44.161182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:44.161182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:44.161182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:44.161182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:44.161182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:44.161182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:44.161182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:44.161182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.161182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:44.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:44.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:44.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:44.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:44.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:44.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:44.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:44.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:44.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:44.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:44.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:44.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:44.161656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:44.161656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:44.161656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:44.161656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.161656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:44.161656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:44.161656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:44.161656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.161656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:44.161656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:44.161656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:44.161656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:44.161656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:44.161656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:44.161656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:44.161656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:44.161656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:44.161656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:44.161656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:44.161955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:44.161955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:44.161955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.161955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:44.165040 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:44.166798 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:44.166886 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:44.167283 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:44.170640 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:44.170811 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:44.173282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.173282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:44.173282 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:44.173282 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:44.173282 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:44.173282 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:44.173282 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:44.173282 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.173282 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:44.173282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.173282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:44.173282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:44.173282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:44.173282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.173282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:44.173604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:44.173604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.173604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:44.173604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:44.173604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:44.173604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.173604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:44.173604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:44.173604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:44.173604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:44.173604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:44.173604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:44.173604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:44.173604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:44.173826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:44.173826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:44.173826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:44.173826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:44.173826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:44.173826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.173826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:44.173826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.173826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:44.173826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:44.173826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:44.173826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:44.173826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:44.174054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:44.174412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:44.174412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:44.174412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:44.174412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:44.174412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:44.174412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:44.174412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:44.174412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:44.174412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:44.174412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:44.174412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:44.174412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:48.601984 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:48.602635 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:48.603044 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:48.604885 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:48.706905 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:48.707108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:48.709044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:48.709044 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:48.709044 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:48.709044 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:48.709044 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:48.709044 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:48.709044 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:48.709044 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:48.710299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.106 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:48.710299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:48.710299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:48.710299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:48.710299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:48.710299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:48.710299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:48.710299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:48.710299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:48.710299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:48.710299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:48.710299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:48.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:48.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:48.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:48.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:48.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:48.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:48.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:48.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:48.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:48.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:48.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:48.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:48.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:48.710944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:48.710944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:48.713304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:48.713304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:48.713304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:48.713304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:48.713304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:48.713304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:48.713304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:48.713304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:48.713304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:48.713304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:48.713304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:48.713304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:48.713304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:48.713304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:48.713304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:48.713304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:48.713630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:48.713630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:48.713630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:48.713630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:48.713630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:48.713630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:48.713630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:48.713630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:48.713630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:48.713630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:48.713630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:48.713630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:48.713630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:48.713630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:48.713630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:48.713630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:48.713630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:48.713630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:48.713630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:48.714003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:48.714003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:48.714003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:49.123741 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:49.124329 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:49.124329 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:49.125978 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:49.126177 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:49.127151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:49.128069 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:49.128186 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:49.128371 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:49.128511 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:49.128649 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:49.128803 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:49.128955 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:49.129098 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:49.129230 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:49.129399 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:49.129531 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:49.131617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.133413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:49.133413 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:49.133413 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:49.133413 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:49.133413 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:49.133413 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:49.133413 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.133413 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:49.135653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.135653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:49.135653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:49.135653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:49.135653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.135653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:49.135653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:49.135653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.135653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:49.135653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:49.135653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:49.135653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.136676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:49.136676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:49.136676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:49.136676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:49.136676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:49.136676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:49.136676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:49.136676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:49.136676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:49.136676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:49.136676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:49.136676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:49.136676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:49.139429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.139429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:49.139429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.139429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:49.139429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:49.139429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:49.139429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:49.139429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:49.139429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.139429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:49.139429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:49.139429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.139429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:49.139429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:49.139429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.139869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:49.139869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:49.139869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.139869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:49.139869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:49.139869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.139869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:49.139869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:49.139869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:49.139869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.139869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:49.139869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:49.139869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:49.139869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:49.139869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:49.139869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:49.139869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:49.139869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:49.139869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:49.140442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:49.140442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:49.140442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:49.140442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:49.140442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.140442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:49.140730 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:49.142479 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:49.142622 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:49.143622 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:24:49.144062 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:49.144284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.145907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:49.145907 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:49.145907 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:49.145907 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:49.145907 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:49.145907 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:49.145907 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.145907 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:49.148303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.148303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:49.148303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:49.148303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:49.148303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.148303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:49.148303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:49.148303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:49.148303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.148303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:49.148303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:49.148303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:49.148303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:49.148303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:49.148303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:49.148812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:49.148812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:49.148812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:49.148812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:49.148812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:49.148812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:49.148812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:49.148812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.148812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:49.149838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.149838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:49.149838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:49.149838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:49.149838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.149838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:49.149838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:49.149838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:24:49.149838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:49.149838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:49.149838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.149838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:49.150207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:49.150207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:49.150207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.150207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:49.150207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:49.150207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:49.150207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:49.150207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:49.150207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:49.150207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:49.150207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:49.150207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:49.150207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:49.150207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:49.150565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:49.150565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:49.150565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.150565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:49.151686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.151686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:49.151686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:49.151686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:49.151686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:49.151686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:49.151686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.151686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:49.151686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:49.151686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.151686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:49.151686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:49.151686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.151686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:49.151686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:49.151686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.155835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:49.155835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:49.155835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:24:49.155835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:49.155835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:49.155835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.155835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:49.155835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:49.155835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:49.155835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.155835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:49.155835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:49.155835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:49.155835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:49.155835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:49.155835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:49.155835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:49.155835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:49.157454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:49.157454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:49.157454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:49.157454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:49.157454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:49.157454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.157454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:49.157454 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:49.158459 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:49.158459 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:49.158459 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:49.158459 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:49.158734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:49.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:49.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:49.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:49.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:49.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:49.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.161388 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:49.163038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.163038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:49.163038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:49.163038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:49.163038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.163038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:49.163038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:49.163038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:49.163038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.163038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:49.163038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:49.163038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:49.163038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:49.163038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:49.163038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:49.163492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:49.163492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:49.163492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:49.163492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:49.163492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:49.163492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:49.163492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:49.163492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.163492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:49.163492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.163492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:49.163492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:49.163492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:49.163492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.164395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:49.164395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:49.164395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:49.164395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:49.164395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.164395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:49.164395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:49.164395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:49.164395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.164395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:49.164395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:49.164395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:49.164395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:49.164798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:49.164798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:49.164798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:49.164798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:49.164798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:49.164798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:49.164798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:49.164798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:49.164798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:49.164798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.164798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:49.173190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.173190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:49.173190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:49.173190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:49.173190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:49.173190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:49.173190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.173190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:49.173190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:49.173190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.173190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:49.173190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:49.173190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.173190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:49.173190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:49.173190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.173190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:49.173929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:49.173929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:49.173929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:49.173929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.173929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:49.173929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:49.173929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:49.173929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.173929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:49.173929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:49.173929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:49.173929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:49.173929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:49.173929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:49.173929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:49.173929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:49.173929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:49.173929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:49.173929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:49.174576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:49.174576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:49.174576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.174576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:49.174576 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:49.175595 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:49.176187 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:49.179475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:49.180943 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:49.181286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:49.181690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.183544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:49.183544 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:49.183544 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:49.183544 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:49.183544 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:49.183544 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:49.183544 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.183544 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:49.192490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.192490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:49.192490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:49.192490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:49.192490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.192490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:49.192490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:49.192490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.192490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:49.192490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:49.192490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:49.192490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.192772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:49.192772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:49.192772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:49.192772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:49.192772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:49.192772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:49.192772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:49.192772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:49.192772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:49.192772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:49.192772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:49.192772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:49.192772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:49.193373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.193373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:49.199480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.199480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:49.199480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:49.199480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:49.199480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:49.199480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:49.199480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.199480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:49.199480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:49.199480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.199480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:49.199480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:49.199480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.199480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:49.199480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:49.199480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:49.199804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:49.199804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:49.199804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.199804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:49.199804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:49.199804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:49.199804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:49.199804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:49.199804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:49.199804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:49.199804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:49.199804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:49.199804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:49.199804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:49.199804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:49.199804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:49.199804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:49.199804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:49.199804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:49.200172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:49.200172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:49.200172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:53.605565 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:53.606999 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:53.607183 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:53.608027 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:53.612266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=240 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:24:53.682637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:53.682637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:53.682637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:53.682637 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:53.682637 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:53.682637 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:53.682637 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:53.682637 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:53.682637 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:53.682637 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:53.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.075 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:53.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:53.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:53.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:53.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:53.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:53.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:53.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:53.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:53.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:53.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:53.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:53.683600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:53.683600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:53.683600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:53.683600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:53.683600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:53.683600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:53.683600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:53.683600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:53.683600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:53.683600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:53.683600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:53.683600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:53.683600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:53.683876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:53.683876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:53.684561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:53.684561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:53.684561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:53.684561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:53.684561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:53.684561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:53.684561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:53.684561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:53.684561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:53.684561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:53.684561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:53.684561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:53.684561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:53.684561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:53.684561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:53.684561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:53.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:53.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:53.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:53.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:53.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:53.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:53.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:53.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:53.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:53.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:53.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:53.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:53.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:53.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:53.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:53.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:53.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:53.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:53.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:53.685491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:53.685491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:53.685491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:54.124188 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:54.124718 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:54.125025 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:54.126895 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:54.127315 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:54.127488 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:54.128489 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:54.128489 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:54.128489 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:54.128489 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:54.128489 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:54.128692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:54.128692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:54.128784 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:54.128913 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:54.129055 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:54.129228 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:54.129419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.129976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:54.129976 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:54.129976 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:54.129976 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:54.129976 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:54.129976 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:54.129976 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.129976 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:54.131103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.131103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:54.131103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:54.131103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:54.131103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.131103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:54.131103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:54.131103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.131103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:54.131103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:54.131103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:54.131103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.131474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:54.131474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:54.131474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:54.131474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:54.131474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:54.131474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:54.131474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:54.131474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:54.131474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:54.131474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:54.131474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:54.131474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:54.131474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:54.131695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.131695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:54.132112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.132112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:54.132112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:54.132112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:54.132112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:54.132112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:54.132112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.132112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:54.132112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:54.132112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.132112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:54.132112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:54.132112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.132112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:54.132112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:54.132112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.132482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:54.132482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:54.132482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.132482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:54.132482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:54.132482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:54.132482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.132482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:54.132482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:54.132482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:54.132482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:54.132482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:54.132482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:54.132482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:54.132482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:54.132482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:54.132482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:54.132482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:54.132482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:54.132810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:54.132810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.132810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:54.147432 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:54.148762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:54.148902 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:54.148970 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:24:54.150437 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:54.150581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.152059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:54.152059 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:54.152059 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:54.152059 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:54.152059 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:54.152059 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:54.152059 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.152059 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:54.153436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.153436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:54.153436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:54.153436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:54.153436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.153436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:54.153436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:54.153436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:54.153436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.153436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:54.153436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:54.153436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:54.153436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:54.153436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:54.153436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:54.153820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:54.153820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:54.153820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:54.153820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:54.153820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:54.153820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:54.153820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:54.153820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.153820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:54.157552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.157552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:54.157552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:54.157552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:54.157552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.157552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:54.157552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:54.157552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:24:54.157552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:54.157552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:54.157552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.157552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:54.158298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:54.158298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:54.158298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.158298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:54.158298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:54.158298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:54.158298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:54.158298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:54.158298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:54.158298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:54.158298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:54.158298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:54.158298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:54.158298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:54.158604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:54.158604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:54.158604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.158604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:54.160941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.160941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:54.160941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:54.160941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:54.160941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:54.160941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:54.160941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.160941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:54.160941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:54.160941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.160941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:54.160941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:54.160941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.160941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:54.160941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:54.160941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.161400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:54.161400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:54.161400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:24:54.161400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:54.161400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:54.161400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.161400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:54.161400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:54.161400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:54.161400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.161400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:54.161400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:54.161400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:54.161400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:54.161400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:54.161400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:54.161400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:54.161400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:54.161762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:54.161762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:54.161762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:54.161762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:54.161762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:54.161762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.161762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:54.161762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:54.161762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:54.161762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:54.164231 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:54.164231 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:54.164231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.164231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:54.164231 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:54.164231 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:54.164231 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:54.164231 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:54.164231 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:54.164231 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.164231 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:54.164231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.164589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:54.164589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:54.164589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:54.164589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.164589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:54.164589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:54.164589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:54.164589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.164589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:54.164589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:54.164589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:54.164589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:54.164589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:54.164589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:54.164589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:54.164589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:54.164589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:54.165306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:54.165306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:54.165306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:54.165306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:54.165306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.165306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:54.165306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.165306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:54.165306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:54.165306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:54.165306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.165306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:54.165306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:54.165687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:54.165687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:54.165687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.165687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:54.165687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:54.165687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:54.165687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.165687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:54.165687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:54.165687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:54.165687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:54.165687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:54.165687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:54.165950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:54.165950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:54.165950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:54.165950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:54.165950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:54.165950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:54.165950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:54.165950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.165950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:54.167557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.167557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:54.167557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:54.167557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:54.167557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:54.167557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:54.167557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.167557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:54.167557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:54.167557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.167557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:54.167557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:54.167557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.167557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:54.167557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:54.167557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.167557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:54.168011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:54.168011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:54.168011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:54.168011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.168011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:54.168011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:54.168011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:54.168011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.168011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:54.168011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:54.168011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:54.168011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:54.168011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:54.168011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:54.168011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:54.168011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:54.168011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:54.168011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:54.168011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:54.168902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:54.168902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:54.168902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.168902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:54.175626 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:54.175811 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:54.175898 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:54.176052 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:54.176329 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:54.176619 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:54.176900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.177638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:54.177638 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:54.177638 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:54.177638 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:54.177638 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:54.177638 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:54.177638 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.177638 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:54.178482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.178482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:54.178482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:54.178482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:54.178482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.178482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:54.178482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:54.178482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.178482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:54.178482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:54.178482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:54.178482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.178796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:54.178796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:54.178796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:54.178796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:54.178796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:54.178796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:54.178796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:54.178796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:54.178796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:54.178796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:54.178796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:54.178796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:54.178796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:54.179038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.179038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:54.181356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.181356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:54.181356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:54.181356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:54.181356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:54.181356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:54.181356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.181356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:54.181356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:54.181356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.181356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:54.181356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:54.181356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.181356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:54.181356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:54.181356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:54.181930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:54.181930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:54.181930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.181930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:54.181930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:54.181930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:54.181930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:54.181930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:54.181930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:54.181930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:54.181930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:54.181930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:54.181930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:54.181930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:54.181930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:54.181930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:54.181930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:54.181930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:54.181930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:54.182536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:54.182536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:54.182536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:58.608633 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:58.609187 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:58.609187 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:58.609829 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:58.653985 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:58.654187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:58.654761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:58.654761 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:58.654761 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:58.654761 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:58.654761 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:58.654761 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:58.654761 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:58.654761 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:58.655278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:58.655278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:58.655278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:58.655278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:58.655278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:58.655278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:58.655278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:58.655278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:58.655278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:58.655278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:58.655278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:58.655278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:58.655550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:58.655550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:58.655550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:58.655550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:58.655550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:58.655550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:58.655550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:58.655550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:58.655550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:58.655550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:58.655550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:58.655550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:58.655550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:58.655764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:58.655764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:58.655959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:58.655959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:58.655959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:58.655959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:58.655959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:58.655959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:58.655959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:58.655959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:58.655959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:58.655959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:58.655959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:58.655959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:58.655959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:58.655959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:58.655959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:58.655959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:58.656286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:24:58.656286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:58.656286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:58.656286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:58.656286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:58.656286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:58.656286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:58.656286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:58.656286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:58.656286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:58.656286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:58.656286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:58.656286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:58.656286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:58.656286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:58.656286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:58.656286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:58.656286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:58.656286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:58.656599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:58.656599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:58.656599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:59.125189 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:59.125731 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:59.125862 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:59.127220 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:59.127375 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:59.127499 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:59.127615 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:59.127689 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:59.127790 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:59.127882 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:59.127987 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:59.128118 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:59.128266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:59.128800 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:59.128800 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:59.128800 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:59.128800 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:59.128800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.129196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:59.129196 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:59.129196 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:59.129196 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:59.129196 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:59.129196 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:59.129196 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.129196 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:59.130117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.130117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:59.130117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:59.130117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:59.130117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.130117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:59.130117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:59.130117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.130117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:59.130117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:59.130117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:59.130117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.130597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:59.130597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:59.130597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:59.130597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:59.130597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:59.130597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:59.130597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:59.130597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:59.130597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:59.130597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:59.130597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:59.130597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:59.130597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:59.130955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.130955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:59.131032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.131032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:59.131032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:59.131032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:59.131032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:59.131032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:59.131032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.131032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:59.131032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:59.131032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.131032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:59.131032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:59.131032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.131032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:59.131032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:59.131032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.131551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:24:59.131551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:59.131551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.131551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:59.131551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:59.131551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:59.131551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.131551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:59.131551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:59.131551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:59.131551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:59.131551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:59.131551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:59.131551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:59.131551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:59.131551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:59.131551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:59.131551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:59.131551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:59.131984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:59.131984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.131984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:59.148922 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:59.149374 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:59.149545 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:59.149702 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:24:59.150071 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:59.150308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.150844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:59.150844 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:59.150844 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:59.150844 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:59.150844 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:59.150844 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:59.150844 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.150844 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:59.151658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.151658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:59.151658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:59.151658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:59.151658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.151658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:59.151658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:59.151658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:59.151658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.151658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:59.151658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:59.151658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:59.151658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:59.151658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:59.151658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:59.151936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:59.151936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:59.151936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:59.151936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:59.151936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:59.151936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:59.151936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:59.151936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.151936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:59.152551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.152551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:59.152551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:59.152551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:59.152551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.152551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:59.152551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:59.152551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:24:59.152551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:59.152551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:59.152551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.152551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:59.152779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:59.152779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:59.152779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.152779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:59.152779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:59.152779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:59.152779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:59.152779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:59.152779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:59.152779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:59.152779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:59.152779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:59.152779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:59.152779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:59.153002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:59.153002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:59.153002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.153002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:59.153701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.153701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:59.153701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:59.153701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:59.153701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:59.153701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:59.153701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.153701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:59.153701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:59.153701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.153701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:59.153701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:59.153701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.153701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:59.153701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:59.153701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.154001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:24:59.154001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:24:59.154001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:24:59.154001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:59.154001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:59.154001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.154001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:59.154001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:59.154001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:59.154001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.154001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:59.154001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:59.154001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:59.154001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:59.154001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:59.154001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:59.154001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:59.154001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:59.154390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:59.154390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:59.154390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:59.154390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:59.154390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:59.154390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.154390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:59.161608 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:59.161889 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:59.162066 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:59.162277 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:59.162579 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:59.162810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.163356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:59.163356 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:59.163356 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:59.163356 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:59.163356 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:59.163356 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:59.163356 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.163356 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:59.164116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.164116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:24:59.164116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:59.164116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:59.164116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.164116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:59.164116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:59.164116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:24:59.164116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.164116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:24:59.164116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:59.164116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:24:59.164116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:59.164116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:24:59.164116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:24:59.164418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:59.164418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:24:59.164418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:59.164418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:24:59.164418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:24:59.164418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:59.164418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:24:59.164418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.164418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:24:59.165060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.165060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:59.165060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:59.165060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:59.165060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.165060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:59.165060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:59.165060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:59.165060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:59.165060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.165060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:59.165272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:59.165272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:59.165272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.165272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:59.165272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:59.165272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:59.165272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:59.165272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:59.165272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:59.165272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:59.165272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:59.165272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:59.165272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:59.165272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:59.165615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:59.165615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:59.165615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.165615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:59.166313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.166313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:59.166313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:59.166313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:59.166313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:59.166313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:59.166313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.166313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:59.166313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:59.166313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.166313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:59.166313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:59.166313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.166313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:59.166313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:59.166313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.166313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:24:59.166681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:24:59.166681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:24:59.166681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:24:59.166681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.166681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:59.166681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:59.166681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:59.166681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.166681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:59.166681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:59.166681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:59.166681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:59.166681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:59.166681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:59.166681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:59.166681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:59.166681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:59.166681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:59.166681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:59.166980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:59.166980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:59.166980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.166980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:24:59.176077 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:24:59.176466 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:24:59.176614 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:24:59.176819 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:24:59.177128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:59.177397 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:24:59.177805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.178360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:24:59.178360 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:59.178360 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:24:59.178360 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:24:59.178360 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:59.178360 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:24:59.178360 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.178360 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:24:59.183143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.183143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:24:59.183143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:59.183143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:24:59.183143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.183143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:59.183143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:59.183143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.183143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:59.183143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:59.183143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:24:59.183143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.183414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:24:59.183414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:59.183414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:24:59.183414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:59.183414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:24:59.183414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:24:59.183414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:59.183414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:24:59.183414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:59.183414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:24:59.183414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:24:59.183414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:59.183414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:24:59.183627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.183627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:24:59.183946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.183946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:24:59.183946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:24:59.183946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:24:59.183946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:24:59.183946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:24:59.183946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.183946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:24:59.183946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:24:59.183946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.183946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:24:59.183946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:24:59.183946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.183946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:24:59.183946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:24:59.183946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:24:59.184233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:24:59.184233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:24:59.184233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.184233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:59.184233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:24:59.184233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:24:59.184233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:24:59.184233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:24:59.184233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:24:59.184233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:24:59.184233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:24:59.184233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:24:59.184233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:24:59.184233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:24:59.184233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:24:59.184233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:24:59.184233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:24:59.184233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:24:59.184233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:24:59.184564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:24:59.184564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:24:59.184564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:03.609405 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:03.609938 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:03.609938 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:03.610505 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:03.651502 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:03.651674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:03.652013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:03.652013 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:03.652013 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:03.652013 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:03.652013 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:03.652013 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:03.652013 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:03.652013 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:03.652930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:03.652930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:03.652930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:03.652930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:03.652930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:03.652930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:03.652930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:03.652930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:03.652930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:03.652930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:03.652930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:03.652930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:03.653333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:03.653333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:03.653333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:03.653333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:03.653333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:03.653333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:03.653333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:03.653333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:03.653333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:03.653333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:03.653333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:03.653333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:03.653333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:03.653706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:03.653706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:03.656004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:03.656004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:03.656004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:03.656004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:03.656004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:03.656004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:03.656004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:03.656004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:03.656004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:03.656004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:03.656004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:03.656004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:03.656004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:03.656004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:03.656004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:03.656004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:03.656622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:03.656622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:03.656622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:03.656622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:03.656622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:03.656622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:03.656622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:03.656622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:03.656622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:03.656622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:03.656622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:03.656622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:03.656622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:03.656622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:03.656622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:03.656622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:03.656622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:03.656622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:03.656622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:03.657191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:03.657191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:03.657191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:04.126624 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:04.127039 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:04.127145 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:04.129281 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:04.129281 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:04.129281 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-5c0116cc-71cb-4460-8774-dc9a047d20bb, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:04.129281 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:04.129281 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:04.129634 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:04.129634 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:04.129634 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:04.129634 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:04.129801 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:04.129878 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:04.129936 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:04.130042 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:04.130137 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:04.130232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.130775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:04.130775 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:04.130775 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:04.130775 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:04.130775 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:04.130775 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:04.130775 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.130775 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:04.131526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.131526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:04.131526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:04.131526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:04.131526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.131526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:04.131526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:04.131526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.131526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:04.131526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:04.131526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:04.131526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.131956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:04.131956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:04.131956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:04.131956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:04.131956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:04.131956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:04.131956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:04.131956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:04.131956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:04.131956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:04.131956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:04.131956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:04.131956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:04.132368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.132368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:04.132368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.132368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:04.132368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:04.132368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:04.132368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:04.132368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:04.132368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.132368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:04.132368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:04.132368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.132368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:04.132368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:04.132368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.132809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:04.132809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:04.132809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.132809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:04.132809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:04.132809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.132809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:04.132809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:04.132809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:04.132809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.132809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:04.132809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:04.132809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:04.132809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:04.132809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:04.132809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:04.132809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:04.132809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:04.132809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:04.133354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:04.133354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:04.133354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:04.133354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:04.133354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.133354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:04.150229 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:04.150229 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:04.150229 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:04.150229 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:25:04.150523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:04.150601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.151344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:04.151344 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:04.151344 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:04.151344 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:04.151344 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:04.151344 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:04.151344 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.151344 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:04.151712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.151712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:04.151712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:04.151712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:04.151712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.151712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:04.151712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:04.151712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:04.151712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.151712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:04.151712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:04.151712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:04.151712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:04.151712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:04.151712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:04.152151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:04.152151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:04.152151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:04.152151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:04.152151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:04.152151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:04.152151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:04.152151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.152151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:04.152402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.152402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:04.152402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:04.152402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:04.152402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.152402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:04.152402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:04.152402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:25:04.152402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:04.152402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:04.152402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.152402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:04.152722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:04.152722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:04.152722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.152722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:04.152722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:04.152722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:04.152722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:04.152722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:04.152722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:04.152722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:04.152722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:04.152722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:04.152722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:04.152722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:04.153089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:04.153089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:04.153089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.153089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:04.153089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.153089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:04.153089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:04.153089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:04.153089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:04.153089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:04.153089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.153089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:04.153089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:04.153089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.153473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:04.153473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:04.153473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.153473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:04.153473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:04.153473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.153473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:04.153473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:04.153473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:25:04.153473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:04.153473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:04.153473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.153473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:04.153473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:04.153473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:04.153473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.153473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:04.153473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:04.153473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:04.155634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:04.155634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:04.155634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:04.155634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:04.155634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:04.155634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:04.155634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:04.155634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:04.155634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:04.155634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:04.155634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.155634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:04.162070 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:04.162609 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:04.163212 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:04.163212 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:04.163412 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:04.163805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.166744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:04.166744 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:04.166744 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:04.166744 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:04.166744 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:04.166744 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:04.166744 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.166744 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:04.166744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.166744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:04.166744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:04.166744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:04.166744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.166744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:04.166744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:04.166744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:04.166744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.166744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:04.167427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:04.167427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:04.167427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:04.167427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:04.167427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:04.167427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:04.167427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:04.167427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:04.167427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:04.167427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:04.167427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:04.167427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:04.167427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.167427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:04.167427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.167848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:04.167848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:04.167848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:04.167848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.167848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:04.167848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:04.167848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:04.167848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:04.167848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.167848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:04.167848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:04.167848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:04.167848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.168084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:04.168084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:04.168084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:04.168084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:04.168084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:04.168084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:04.168084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:04.168084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:04.168084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:04.168084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:04.168084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:04.168084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:04.168084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:04.168345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.168345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:04.168345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.168345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:04.168345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:04.168345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:04.168345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:04.168345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:04.168345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.168345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:04.168345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:04.168345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.168345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:04.168345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:04.168345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.168606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:04.168606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:04.168606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.168606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:04.168606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:04.168606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:04.168606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:04.168606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.168606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:04.168606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:04.168606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:04.168606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.168606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:04.168606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:04.168606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:04.168606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:04.168606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:04.168606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:04.168606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:04.168966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:04.168966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:04.168966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:04.168966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:04.168966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:04.168966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:04.168966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.168966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:04.176988 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:04.178742 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:04.178742 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:04.178742 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:04.178742 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:04.179174 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:04.179351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.181813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:04.181813 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:04.181813 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:04.181813 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:04.181813 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:04.181813 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:04.181813 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.181813 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:04.181813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.181813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:04.181813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:04.181813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:04.181813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.181813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:04.181813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:04.183062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.183062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:04.183062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:04.183062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:04.183062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.183062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:04.183062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:04.183062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:04.183062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:04.183062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:04.183062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:04.183062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:04.183062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:04.183062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:04.184130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:04.184130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:04.184130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:04.184130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:04.184130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.184130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:04.184130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.184130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:04.184130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:04.184130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:04.184130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:04.184130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:04.184130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.184130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:04.185341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:04.187685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:04.187685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:04.187685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:04.187685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:04.187685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:04.187685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:04.187685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:04.187685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:04.187685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:04.187685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:08.609851 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:08.610450 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:08.610450 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:08.610968 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:08.665941 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:08.666068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:08.666743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:08.666743 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:08.666743 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:08.666743 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:08.666743 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:08.666743 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:08.666743 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:08.666743 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:08.667476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:08.667476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:08.667476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:08.667476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:08.667476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:08.667476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:08.667476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:08.667476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:08.667476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:08.667476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:08.667476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:08.667476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:08.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:08.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:08.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:08.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:08.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:08.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:08.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:08.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:08.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:08.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:08.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:08.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:08.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:08.668970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:08.668970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:08.668970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:08.668970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:08.668970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:08.668970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:08.668970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:08.668970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:08.668970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:08.668970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:08.668970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:08.668970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:08.668970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:08.668970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:08.668970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:08.669300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:08.669300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:08.669300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:08.669300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:08.669300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:08.669300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:08.669300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:08.669300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:08.669300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:08.669300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:08.669300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:08.669300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:08.669300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:08.669300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:08.669300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:08.669300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:08.669300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:08.669300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:08.669300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:08.669712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:08.669712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:08.669712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:08.669712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:08.669712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:08.669712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:09.127110 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:09.127434 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:09.127564 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:09.129082 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:09.129283 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:09.129450 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:09.129580 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:09.129696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:09.129794 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:09.129892 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:09.130404 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:09.130404 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:09.130404 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:09.130404 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:09.130571 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:09.130571 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:09.130634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.131335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:09.131335 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:09.131335 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:09.131335 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:09.131335 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:09.131335 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:09.131335 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.131335 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:09.132534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.132534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:09.132534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:09.132534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:09.132534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.132534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:09.132534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:09.132534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.132534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:09.132534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:09.132534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:09.132534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.132827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:09.132827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:09.132827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:09.132827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:09.132827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:09.132827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:09.132827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:09.132827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:09.132827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:09.132827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:09.132827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:09.132827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:09.132827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:09.133082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.133082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:09.133848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.133848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:09.133848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:09.133848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:09.133848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:09.133848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:09.133848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.133848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:09.133848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:09.133848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.133848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:09.133848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:09.133848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.133848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:09.133848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:09.133848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.134223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:09.134223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:09.134223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.134223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:09.134223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:09.134223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:09.134223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.134223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:09.134223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:09.134223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:09.134223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:09.134223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:09.134223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:09.134223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:09.134223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:09.134223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:09.134223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:09.134223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:09.134223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:09.134815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:09.134815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.134815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:09.149940 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:09.151614 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:09.151741 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:09.151869 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:25:09.155687 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:09.155887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.157002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:09.157002 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:09.157002 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:09.157002 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:09.157002 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:09.157002 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:09.157002 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.157002 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:09.157737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.157737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:09.157737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:09.157737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:09.157737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.157737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:09.157737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:09.157737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:09.157737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.157737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:09.157737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:09.157737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:09.157737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:09.157737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:09.157737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:09.158010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:09.158010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:09.158010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:09.158010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:09.158010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:09.158010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:09.158010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:09.158010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.158010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:09.159673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.159673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:09.159673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:09.159673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:09.159673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.159673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:09.159673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:09.159673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:25:09.159673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:09.159673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:09.159673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.159673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:09.160210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:09.160210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:09.160210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.160210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:09.160210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:09.160210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:09.160210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:09.160210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:09.160210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:09.160210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:09.160210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:09.160210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:09.160210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:09.160210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:09.160627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:09.160627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:09.160627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.160627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:09.160627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.160627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:09.160627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:09.160627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:09.160627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:09.160627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:09.160627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.160627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:09.160627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:09.160627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:09.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:09.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:09.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:09.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:09.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:09.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:25:09.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:09.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:09.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:09.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:09.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:09.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:09.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:09.161072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:09.161576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:09.161576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:09.161576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:09.161576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:09.161576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:09.161576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:09.161576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:09.161576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:09.161576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:09.161576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:09.161576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.161576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:09.162475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:09.164471 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:09.164566 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:09.164672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:09.166331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:09.166526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.168832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:09.168832 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:09.168832 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:09.168832 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:09.168832 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:09.168832 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:09.168832 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.168832 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:09.169601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.169601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:09.169601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:09.169601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:09.169601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.169601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:09.169601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:09.169601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:09.169601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.169601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:09.169601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:09.169601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:09.169601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:09.169601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:09.169601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:09.170088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:09.170088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:09.170088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:09.170088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:09.170088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:09.170088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:09.170088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:09.170088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.170088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:09.170396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.170396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:09.170396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:09.170396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:09.170396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.170396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:09.170396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:09.170396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:09.170396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:09.170396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.170396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:09.170746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:09.170746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:09.170746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.170746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:09.170746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:09.170746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:09.170746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:09.170746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:09.170746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:09.170746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:09.170746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:09.170746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:09.170746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:09.170746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:09.171144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:09.171144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:09.171144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.171144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:09.171144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.171144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:09.171144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:09.171144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:09.171144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:09.171144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:09.171144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.171144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:09.171144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:09.171144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.171579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:09.171579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:09.171579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.171579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:09.171579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:09.171579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.171579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:09.171579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:09.171579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:09.171579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:09.171579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.171579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:09.171579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:09.171579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:09.171579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.171579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:09.171579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:09.171579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:09.171579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:09.172056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:09.172056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:09.172056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:09.172056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:09.172056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:09.172056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:09.172056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:09.172056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:09.172056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:09.172056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.172056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:09.177338 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:09.177655 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:09.177768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:09.177926 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:09.179480 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:09.180057 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:09.180057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.180401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:09.180401 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:09.180401 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:09.180401 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:09.180401 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:09.180401 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:09.180401 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.180401 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:09.181172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.181172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:09.181172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:09.181172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:09.181172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.181172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:09.181172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:09.181172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.181172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:09.181172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:09.181172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:09.181172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.181571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:09.181571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:09.181571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:09.181571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:09.181571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:09.181571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:09.181571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:09.181571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:09.181571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:09.181571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:09.181571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:09.181571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:09.181571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:09.181873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.181873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:09.181873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.181873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:09.181873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:09.181873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:09.181873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:09.181873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:09.181873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.181873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:09.181873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:09.181873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.181873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:09.181873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:09.181873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.182353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:09.182353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:09.182353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:09.182353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:09.182353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:09.182353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.182353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:09.182353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:09.182353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:09.182353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:09.182353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:09.182353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:09.182353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:09.182353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:09.182353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:09.182353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:09.182353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:09.182353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:09.182353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:09.182806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:09.182806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:09.182806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:09.182806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:09.182806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:09.182806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:13.610349 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:13.610948 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:13.610948 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:13.611927 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:13.664755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:13.664984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:13.665782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:13.665782 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:13.665782 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:13.665782 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:13.665782 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:13.665782 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:13.665782 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:13.665782 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:13.666611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:13.666611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:13.666611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:13.666611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:13.666611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:13.666611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:13.666611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:13.666611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:13.666611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:13.666611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:13.666611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:13.666611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:13.666860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:13.666860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:13.666860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:13.666860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:13.666860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:13.666860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:13.666860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:13.666860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:13.666860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:13.666860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:13.666860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:13.666860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:13.666860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:13.667119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:13.667119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:13.667713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:13.667713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:13.667713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:13.667713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:13.667713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:13.667713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:13.667713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:13.667713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:13.667713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:13.667713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:13.667713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:13.667713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:13.667713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:13.667713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:13.667713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:13.667713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:13.668115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:13.668115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:13.668115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:13.668115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:13.668115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:13.668115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:13.668115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:13.668115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:13.668115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:13.668115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:13.668115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:13.668115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:13.668115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:13.668115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:13.668115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:13.668115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:13.668115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:13.668115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:13.668115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:13.668503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:13.668503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:13.668503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:14.127633 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:14.127932 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:14.128020 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:14.129279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:14.129418 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:14.129558 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:14.129851 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:14.129928 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:14.130037 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:14.130146 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:14.130268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:14.130419 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:14.130549 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:14.130672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:14.130797 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:14.130915 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:14.131052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.131933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:14.131933 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:14.131933 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:14.131933 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:14.131933 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:14.131933 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:14.131933 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.131933 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:14.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:14.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:14.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:14.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:14.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:14.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:14.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:14.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:14.132615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.132915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:14.132915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:14.132915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:14.132915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:14.132915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:14.132915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:14.132915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:14.132915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:14.132915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:14.132915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:14.132915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:14.132915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:14.132915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:14.133219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.133219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:14.133594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.133594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:14.133594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:14.133594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:14.133594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:14.133594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:14.133594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.133594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:14.133594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:14.133594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.133594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:14.133594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:14.133594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.133594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:14.133594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:14.133594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.133898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:14.133898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:14.133898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.133898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:14.133898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:14.133898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:14.133898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.133898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:14.133898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:14.133898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:14.133898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:14.133898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:14.133898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:14.133898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:14.133898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:14.133898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:14.133898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:14.133898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:14.133898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:14.134244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:14.134244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.134244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:14.151829 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:14.152008 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:14.152118 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:14.152265 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:25:14.152559 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:14.152730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.153139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:14.153139 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:14.153139 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:14.153139 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:14.153139 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:14.153139 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:14.153139 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.153139 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:14.153812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.153812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:14.153812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:14.153812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:14.153812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.153812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:14.153812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:14.153812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:14.153812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.153812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:14.153812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:14.153812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:14.153812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:14.153812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:14.153812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:14.154155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:14.154155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:14.154155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:14.154155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:14.154155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:14.154155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:14.154155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:14.154155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.154155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:14.154621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.154621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:14.154621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:14.154621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:14.154621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.154621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:14.154621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:14.154621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:25:14.154621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:14.154621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:14.154621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.154621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:14.154951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:14.154951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:14.154951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.154951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:14.154951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:14.154951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:14.154951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:14.154951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:14.154951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:14.154951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:14.154951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:14.154951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:14.154951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:14.154951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:14.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:14.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:14.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:14.155553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.155553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:14.155553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:14.155553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:14.155553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:14.155553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:14.155553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.155553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:14.155553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:14.155553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.155553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:14.155553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:14.155553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.155553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:14.155553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:14.155553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.155839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:14.155839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:14.155839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:25:14.155839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:14.155839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:14.155839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.155839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:14.155839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:14.155839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:14.155839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.155839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:14.155839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:14.155839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:14.155839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:14.155839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:14.155839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:14.155839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:14.155839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:14.156181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:14.156181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:14.156181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:14.156181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:14.156181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:14.156181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.156181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:14.164619 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:14.164929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:14.164989 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:14.165102 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:14.165371 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:14.165592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.165918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:14.165918 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:14.165918 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:14.165918 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:14.165918 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:14.165918 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:14.165918 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.165918 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:14.166494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.166494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:14.166494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:14.166494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:14.166494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.166494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:14.166494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:14.166494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:14.166494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.166494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:14.166494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:14.166494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:14.166494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:14.166494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:14.166494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:14.166773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:14.166773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:14.166773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:14.166773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:14.166773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:14.166773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:14.166773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:14.166773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.166773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:14.167088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.167088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:14.167088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:14.167088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:14.167088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.167088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:14.167088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:14.167088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:14.167088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:14.167088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.167088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:14.167458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:14.167458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:14.167458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.167458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:14.167458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:14.167458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:14.167458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:14.167458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:14.167458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:14.167458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:14.167458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:14.167458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:14.167458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:14.167458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:14.167851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:14.167851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:14.167851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.167851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:14.168294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.168294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:14.168294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:14.168294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:14.168294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:14.168294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:14.168294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.168294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:14.168294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:14.168294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.168294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:14.168294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:14.168294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.168294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:14.168294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:14.168294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.168294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:14.168847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:14.168847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:14.168847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:14.168847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.168847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:14.168847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:14.168847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:14.168847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.168847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:14.168847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:14.168847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:14.168847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:14.168847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:14.168847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:14.168847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:14.168847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:14.168847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:14.168847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:14.168847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:14.169706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:14.169706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:14.169706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.169706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:14.177756 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:14.178012 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:14.178138 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:14.178303 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:14.178571 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:14.178770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:14.179062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.179458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:14.179458 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:14.179458 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:14.179458 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:14.179458 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:14.179458 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:14.179458 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.179458 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:14.180127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.180127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:14.180127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:14.180127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:14.180127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.180127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:14.180127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:14.180127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.180127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:14.180127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:14.180127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:14.180127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.180449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:14.180449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:14.180449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:14.180449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:14.180449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:14.180449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:14.180449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:14.180449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:14.180449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:14.180449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:14.180449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:14.180449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:14.180449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:14.180650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.180650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:14.180977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.180977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:14.180977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:14.180977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:14.180977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:14.180977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:14.180977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.180977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:14.180977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:14.180977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.180977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:14.180977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:14.180977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.180977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:14.180977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:14.180977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:14.181340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:14.181340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:14.181340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.181340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:14.181340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:14.181340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:14.181340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:14.181340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:14.181340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:14.181340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:14.181340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:14.181340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:14.181340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:14.181340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:14.181340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:14.181340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:14.181340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:14.181340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:14.181340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:14.181674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:14.181674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:14.181674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:18.611589 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:18.612128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:18.612128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:18.614266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:18.659497 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:18.659723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:18.660279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:18.660279 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:18.660279 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:18.660279 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:18.660279 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:18.660279 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:18.660279 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:18.660279 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:18.661201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:18.661201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:18.661201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:18.661201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:18.661201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:18.661201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:18.661201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:18.661201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:18.661201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:18.661201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:18.661201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:18.661201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:18.661616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:18.661616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:18.661616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:18.661616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:18.661616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:18.661616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:18.661616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:18.661616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:18.661616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:18.661616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:18.661616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:18.661616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:18.661616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:18.662069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:18.662069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:18.662354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:18.662354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:18.662354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:18.662354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:18.662354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:18.662354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:18.662354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:18.662354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:18.662354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:18.662354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:18.662354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:18.662354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:18.662354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:18.662354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:18.662354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:18.662354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:18.662683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:18.662683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:18.662683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:18.662683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:18.662683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:18.662683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:18.662683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:18.662683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:18.662683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:18.662683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:18.662683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:18.662683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:18.662683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:18.662683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:18.662683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:18.662683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:18.662683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:18.662683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:18.662683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:18.663025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:18.663025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:18.663025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:19.128175 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:19.128562 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:19.128685 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:19.129908 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:19.130070 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:19.130154 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:19.130292 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:19.130413 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:19.130553 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:19.130663 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:19.130777 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:19.130872 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:19.130964 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:19.131067 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:19.131163 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:19.131266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:19.131498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.131930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:19.131930 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:19.131930 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:19.131930 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:19.131930 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:19.131930 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:19.131930 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.131930 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:19.132586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.132586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:19.132586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:19.132586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:19.132586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.132586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:19.132586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:19.132586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.132586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:19.132586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:19.132586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:19.132586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.132987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:19.132987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:19.132987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:19.132987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:19.132987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:19.132987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:19.132987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:19.132987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:19.132987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:19.132987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:19.132987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:19.132987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:19.132987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:19.133218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.133218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:19.133430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.133430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:19.133430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:19.133430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:19.133430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:19.133430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:19.133430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.133430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:19.133430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:19.133430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.133430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:19.133430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:19.133430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.133430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:19.133430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:19.133430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.133747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:19.133747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:19.133747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.133747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:19.133747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:19.133747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:19.133747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.133747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:19.133747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:19.133747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:19.133747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:19.133747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:19.133747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:19.133747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:19.133747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:19.133747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:19.133747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:19.133747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:19.133747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:19.134078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:19.134078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.134078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:19.152137 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:19.152498 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:19.152603 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:19.152717 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:25:19.152993 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:19.153186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.153582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:19.153582 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:19.153582 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:19.153582 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:19.153582 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:19.153582 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:19.153582 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.153582 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:19.154105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.154105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:19.154105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:19.154105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:19.154105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.154105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:19.154105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:19.154105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:19.154105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.154105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:19.154105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:19.154105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:19.154105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:19.154105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:19.154105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:19.154396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:19.154396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:19.154396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:19.154396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:19.154396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:19.154396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:19.154396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:19.154396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.154396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:19.154696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.154696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:19.154696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:19.154696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:19.154696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.154696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:19.154696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:19.154696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:25:19.154696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:19.154696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:19.154696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.154696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:19.154910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:19.154910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:19.154910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.154910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:19.154910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:19.154910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:19.154910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:19.154910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:19.154910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:19.154910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:19.154910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:19.154910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:19.154910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:19.154910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:19.155158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:19.155158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:19.155158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.155158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:19.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:19.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:19.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:19.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:19.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:19.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:19.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:19.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:19.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:19.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:19.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:19.155438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.155737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:19.155737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:19.155737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:25:19.155737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:19.155737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:19.155737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.155737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:19.155737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:19.155737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:19.155737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.155737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:19.155737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:19.155737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:19.155737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:19.155737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:19.155737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:19.155737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:19.155737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:19.156025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:19.156025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:19.156025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:19.156025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:19.156025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:19.156025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.156025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:19.164950 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:19.165207 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:19.165352 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:19.165514 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:19.165836 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:19.166103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.166643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:19.166643 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:19.166643 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:19.166643 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:19.166643 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:19.166643 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:19.166643 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.166643 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:19.167446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.167446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:19.167446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:19.167446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:19.167446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.167446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:19.167446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:19.167446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:19.167446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.167446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:19.167446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:19.167446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:19.167446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:19.167446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:19.167446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:19.167705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:19.167705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:19.167705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:19.167705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:19.167705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:19.167705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:19.167705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:19.167705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.167705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:19.168445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.168445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:19.168445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:19.168445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:19.168445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.168445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:19.168445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:19.168445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:19.168445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:19.168445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.168445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:19.168674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:19.168674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:19.168674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.168674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:19.168674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:19.168674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:19.168674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:19.168674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:19.168674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:19.168674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:19.168674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:19.168674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:19.168674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:19.168674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:19.168924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:19.168924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:19.168924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.168924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:19.169588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.169588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:19.169588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:19.169588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:19.169588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:19.169588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:19.169588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.169588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:19.169588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:19.169588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.169588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:19.169588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:19.169588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.169588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:19.169588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:19.169588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.169588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:19.169966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:19.169966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:19.169966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:19.169966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.169966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:19.169966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:19.169966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:19.169966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.169966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:19.169966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:19.169966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:19.169966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:19.169966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:19.169966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:19.169966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:19.169966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:19.169966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:19.169966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:19.169966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:19.170325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:19.170325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:19.170325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.170325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:19.178155 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:19.178378 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:19.178524 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:19.178701 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:19.178985 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:19.179298 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:19.179865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.180403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:19.180403 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:19.180403 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:19.180403 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:19.180403 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:19.180403 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:19.180403 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.180403 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:19.181315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.181315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:19.181315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:19.181315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:19.181315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.181315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:19.181315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:19.181315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.181315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:19.181315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:19.181315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:19.181315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.181546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:19.181546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:19.181546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:19.181546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:19.181546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:19.181546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:19.181546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:19.181546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:19.181546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:19.181546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:19.181546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:19.181546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:19.181546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:19.181778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.181778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:19.182463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.182463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:19.182463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:19.182463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:19.182463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:19.182463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:19.182463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.182463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:19.182463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:19.182463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.182463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:19.182463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:19.182463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.182463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:19.182463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:19.182463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:19.182727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:19.182727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:19.182727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.182727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:19.182727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:19.182727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:19.182727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:19.182727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:19.182727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:19.182727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:19.182727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:19.182727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:19.182727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:19.182727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:19.182727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:19.182727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:19.182727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:19.182727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:19.182727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:19.183091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:19.183091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:19.183091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:23.612011 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:23.612433 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:23.612523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:23.613965 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:23.658152 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:23.658383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:23.658802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:23.658802 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:23.658802 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:23.658802 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:23.658802 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:23.658802 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:23.658802 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:23.658802 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:23.659504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:23.659504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:23.659504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:23.659504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:23.659504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:23.659504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:23.659504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:23.659504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:23.659504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:23.659504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:23.659504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:23.659504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:23.659758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:23.659758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:23.659758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:23.659758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:23.659758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:23.659758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:23.659758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:23.659758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:23.659758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:23.659758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:23.659758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:23.659758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:23.659758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:23.660076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:23.660076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:23.660248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:23.660248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:23.660248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:23.660248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:23.660248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:23.660248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:23.660248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:23.660248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:23.660248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:23.660248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:23.660248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:23.660248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:23.660248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:23.660248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:23.660248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:23.660248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:23.660592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:23.660592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:23.660592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:23.660592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:23.660592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:23.660592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:23.660592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:23.660592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:23.660592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:23.660592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:23.660592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:23.660592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:23.660592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:23.660592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:23.660592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:23.660592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:23.660592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:23.660592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:23.660592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:23.660974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:23.660974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:23.660974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:24.130161 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:24.130973 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:24.130973 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:24.132363 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:24.133305 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:24.133305 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:24.133305 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:24.133305 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:24.133305 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:24.133539 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:24.133539 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:24.135114 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:24.135114 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:24.135114 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:24.135114 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:24.135114 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:24.135114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.135397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:24.135397 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:24.135397 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:24.135397 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:24.135397 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:24.135397 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:24.135397 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.135397 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:24.136528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.136528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:24.136528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:24.136528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:24.136528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.136528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:24.136528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:24.136528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.136528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:24.136528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:24.136528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:24.136528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.136959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:24.136959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:24.136959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:24.136959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:24.136959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:24.136959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:24.136959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:24.136959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:24.136959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:24.136959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:24.136959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:24.136959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:24.136959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:24.137351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.137351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:24.137351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.137351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:24.137351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:24.137351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:24.137351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:24.137351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:24.137351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.137351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:24.137351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:24.137351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.137351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:24.137351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:24.137351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.137802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:24.137802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:24.137802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.137802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:24.137802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:24.137802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.137802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:24.137802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:24.137802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:24.137802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.137802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:24.137802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:24.137802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:24.137802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:24.137802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:24.137802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:24.137802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:24.137802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:24.137802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:24.138355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:24.138355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:24.138355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:24.138355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:24.138355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.138355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:24.152677 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:24.153393 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:24.153494 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:24.155065 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:25:24.155524 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:24.155601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.156413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:24.156413 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:24.156413 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:24.156413 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:24.156413 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:24.156413 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:24.156413 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.156413 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:24.157054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.157054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:24.157054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:24.157054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:24.157054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.157054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:24.157054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:24.157054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:24.157054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.157054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:24.157054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:24.157054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:24.157054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:24.157054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:24.157054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:24.157589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:24.157589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:24.157589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:24.157589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:24.157589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:24.157589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:24.157589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:24.157589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.157589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:24.157833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.157833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:24.157833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:24.157833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:24.157833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.157833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:24.157833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:24.157833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:25:24.157833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:24.157833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:24.157833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.157833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:24.158087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:24.158087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:24.158087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.158087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:24.158087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:24.158087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:24.158087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:24.158087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:24.158087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:24.158087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:24.158087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:24.158087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:24.158087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:24.158087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:24.158407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:24.158407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:24.158407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.158407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:24.158407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.158407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:24.158407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:24.158407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:24.158407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:24.158407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:24.158407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.158407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:24.158407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:24.158407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.158698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:24.158698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:24.158698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.158698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:24.158698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:24.158698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.158698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:24.158698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:24.158698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:25:24.158698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:24.158698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:24.158698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.158698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:24.158698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:24.158698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:24.158698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.158698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:24.158698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:24.158698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:24.159170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:24.159170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:24.159170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:24.159170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:24.159170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:24.159170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:24.159170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:24.159170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:24.159170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:24.159170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:24.159170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.159170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:24.165337 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:24.165803 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:24.165931 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:24.166051 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:24.166332 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:24.166645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.167121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:24.167121 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:24.167121 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:24.167121 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:24.167121 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:24.167121 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:24.167121 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.167121 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:24.167613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.167613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:24.167613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:24.167613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:24.167613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.167613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:24.167613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:24.167613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:24.167613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.167613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:24.167613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:24.167613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:24.167613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:24.167613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:24.167613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:24.167904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:24.167904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:24.167904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:24.167904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:24.167904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:24.167904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:24.167904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:24.167904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.167904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:24.168860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.168860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:24.168860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:24.168860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:24.168860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.168860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:24.168860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:24.168860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:24.168860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:24.168860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.168860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:24.169134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:24.169134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:24.169134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.169134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:24.169134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:24.169134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:24.169134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:24.169134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:24.169134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:24.169134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:24.169134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:24.169134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:24.169134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:24.169134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:24.169430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:24.169430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:24.169430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.169430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:24.169430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.169430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:24.169430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:24.169430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:24.169430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:24.169430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:24.169430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.169430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:24.169430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:24.169430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.169737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:24.169737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:24.169737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.169737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:24.169737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:24.169737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.169737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:24.169737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:24.169737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:24.169737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:24.169737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.169737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:24.169737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:24.169737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:24.169737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.169737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:24.169737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:24.169737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:24.169737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:24.170105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:24.170105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:24.170105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:24.170105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:24.170105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:24.170105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:24.170105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:24.170105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:24.170105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:24.170105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.170105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:24.178589 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:24.185946 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:24.186154 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:24.186373 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:24.186693 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:24.186955 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:24.187318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.188456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:24.188456 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:24.188456 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:24.188456 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:24.188456 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:24.188456 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:24.188456 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.188456 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:24.189571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.189571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:24.189571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:24.189571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:24.189571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.189571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:24.189571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:24.189571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.189571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:24.189571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:24.189571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:24.189571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.189846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:24.189846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:24.189846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:24.189846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:24.189846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:24.189846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:24.189846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:24.189846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:24.189846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:24.189846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:24.189846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:24.189846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:24.189846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:24.190192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.190192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:24.191266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.191266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:24.191266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:24.191266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:24.191266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:24.191266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:24.191266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.191266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:24.191266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:24.191266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.191266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:24.191266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:24.191266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.191266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:24.191266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:24.191266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:24.191649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:24.191649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:24.191649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.191649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:24.191649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:24.191649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:24.191649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:24.191649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:24.191649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:24.191649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:24.191649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:24.191649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:24.191649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:24.191649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:24.191649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:24.191649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:24.191649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:24.191649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:24.191649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:24.192098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:24.192098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:24.192098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:28.612506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:28.613734 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:28.613849 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:28.614608 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:28.667577 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:28.667898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:28.669288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:28.669288 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:28.669288 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:28.669288 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:28.669288 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:28.669288 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:28.669288 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:28.669288 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:28.670224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:28.670224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:28.670224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:28.670224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:28.670224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:28.670224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:28.670224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:28.670224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:28.670224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:28.670224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:28.670224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:28.670224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:28.670831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:28.670831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:28.670831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:28.670831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:28.670831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:28.670831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:28.670831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:28.670831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:28.670831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:28.670831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:28.670831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:28.670831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:28.670831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:28.671393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:28.671393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:28.671393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:28.671393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:28.671393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:28.671393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:28.671393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:28.671393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:28.671393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:28.671393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:28.671393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:28.671393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:28.671393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:28.671393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:28.671393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:28.671962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:28.671962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:28.671962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:28.671962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:28.671962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:28.671962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:28.671962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:28.671962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:28.671962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:28.671962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:28.671962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:28.671962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:28.671962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:28.671962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:28.671962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:28.671962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:28.671962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:28.671962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:28.671962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:28.672619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:28.672619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:28.672619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:28.672619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:28.672619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:28.672619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:29.130768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:29.133055 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:29.133186 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:29.136163 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:29.136163 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:29.136163 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:29.136163 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:29.136163 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:29.136692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:29.136692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:29.136692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:29.136692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:29.136692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:29.136880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:29.136880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:29.136880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:29.136880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.136880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:29.136880 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:29.136880 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:29.136880 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:29.136880 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:29.136880 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:29.136880 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.136880 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:29.137480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.137480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:29.137480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:29.137480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:29.137480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.137480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:29.137480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:29.137480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.137480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:29.137480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:29.137480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:29.137480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.137837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:29.137837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:29.137837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:29.137837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:29.137837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:29.137837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:29.137837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:29.137837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:29.137837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:29.137837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:29.137837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:29.137837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:29.137837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:29.138227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.138227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:29.138540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.138540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:29.138540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:29.138540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:29.138540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:29.138540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:29.138540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.138540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:29.138540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:29.138540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.138540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:29.138540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:29.138540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.138540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:29.138540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:29.138540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.139078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:29.139078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:29.139078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.139078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:29.139078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:29.139078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:29.139078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.139078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:29.139078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:29.139078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:29.139078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:29.139078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:29.139078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:29.139078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:29.139078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:29.139078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:29.139078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:29.139078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:29.139078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:29.139659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:29.139659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.139659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:29.155548 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:29.155962 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:29.156078 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:29.156190 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:25:29.157518 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:29.157518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.157685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:29.157685 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:29.157685 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:29.157685 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:29.157685 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:29.157685 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:29.157685 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.157685 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:29.158530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.158530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:29.158530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:29.158530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:29.158530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.158530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:29.158530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:29.158530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:29.158530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.158530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:29.158530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:29.158530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:29.158530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:29.158530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:29.158530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:29.159030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:29.159030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:29.159030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:29.159030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:29.159030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:29.159030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:29.159030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:29.159030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.159030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:29.159418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.159418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:29.159418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:29.159418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:29.159418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.159418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:29.159418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:29.159418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:25:29.159418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:29.159418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:29.159418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.159418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:29.159790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:29.159790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:29.159790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.159790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:29.159790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:29.159790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:29.159790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:29.159790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:29.159790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:29.159790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:29.159790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:29.159790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:29.159790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:29.159790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:29.160218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:29.160218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:29.160218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.160218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:29.160218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.160218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:29.160218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:29.160218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:29.160218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:29.160218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:29.160218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.160218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:29.160218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:29.160218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.160650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:29.160650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:29.160650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.160650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:29.160650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:29.160650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.160650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:29.160650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:29.160650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:25:29.160650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:29.160650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:29.160650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.160650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:29.160650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:29.160650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:29.160650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.160650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:29.160650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:29.160650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:29.161213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:29.161213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:29.161213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:29.161213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:29.161213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:29.161213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:29.161213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:29.161213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:29.161213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:29.161213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:29.161213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.161213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:29.165779 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:29.166879 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:29.167409 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:29.167409 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:29.167760 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:29.168211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.168813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:29.168813 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:29.168813 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:29.168813 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:29.168813 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:29.168813 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:29.168813 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.168813 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:29.170391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.170391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:29.170391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:29.170391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:29.170391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.170391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:29.170391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:29.170391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:29.170391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.170391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:29.170391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:29.170391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:29.170391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:29.170391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:29.170391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:29.170792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:29.170792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:29.170792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:29.170792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:29.170792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:29.170792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:29.170792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:29.170792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.170792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:29.170792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.170792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:29.170792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:29.170792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:29.170792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.171212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:29.171212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:29.171212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:29.171212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:29.171212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.171212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:29.171212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:29.171212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:29.171212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.171212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:29.171212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:29.171212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:29.171212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:29.171587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:29.171587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:29.171587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:29.171587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:29.171587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:29.171587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:29.171587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:29.171587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:29.171587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:29.171587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.171587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:29.173204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.173204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:29.173204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:29.173204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:29.173204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:29.173204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:29.173204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.173204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:29.173204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:29.173204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.173204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:29.173204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:29.173204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.173204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:29.173204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:29.173204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.173204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:29.174064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:29.174064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:29.174064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:29.174064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.174064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:29.174064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:29.174064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:29.174064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.174064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:29.174064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:29.174064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:29.174064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:29.174064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:29.174064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:29.174064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:29.174064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:29.174064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:29.174064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:29.174064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:29.175289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:29.175289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:29.175289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.175289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:29.185921 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:29.186840 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:29.186840 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:29.186840 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:29.187520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:29.188404 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:29.189589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.189860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:29.189860 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:29.189860 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:29.189860 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:29.189860 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:29.189860 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:29.189860 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.189860 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:29.191210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.191210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:29.191210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:29.191210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:29.191210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.191210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:29.191210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:29.191210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.191210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:29.191210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:29.191210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:29.191210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.191647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:29.191647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:29.191647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:29.191647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:29.191647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:29.191647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:29.191647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:29.191647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:29.191647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:29.191647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:29.191647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:29.191647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:29.191647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:29.192064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.192064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:29.194778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.194778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:29.194778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:29.194778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:29.194778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:29.194778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:29.194778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.194778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:29.194778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:29.194778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.194778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:29.194778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:29.194778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.194778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:29.194778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:29.194778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:29.195232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:29.195232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:29.195232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.195232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:29.195232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:29.195232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:29.195232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:29.195232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:29.195232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:29.195232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:29.195232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:29.195232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:29.195232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:29.195232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:29.195232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:29.195232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:29.195232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:29.195232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:29.195232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:29.195741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:29.195741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:29.195741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:33.613882 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:33.614566 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:33.614680 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:33.615358 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:33.680974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:33.680974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:33.681451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:33.681451 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:33.681451 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:33.681451 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:33.681451 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:33.681451 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:33.681451 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:33.681451 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:33.682552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.067 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:33.682552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:33.682552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:33.682552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:33.682552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:33.682552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:33.682552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:33.682552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:33.682552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:33.682552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:33.682552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:33.682552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:33.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:33.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:33.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:33.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:33.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:33.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:33.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:33.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:33.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:33.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:33.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:33.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:33.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:33.683389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:33.683389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:33.683973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:33.683973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:33.683973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:33.683973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:33.683973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:33.683973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:33.683973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:33.683973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:33.683973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:33.683973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:33.683973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:33.683973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:33.683973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:33.683973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:33.683973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:33.683973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:33.684562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:33.684562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:33.684562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:33.684562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:33.684562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:33.684562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:33.684562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:33.684562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:33.684562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:33.684562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:33.684562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:33.684562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:33.684562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:33.684562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:33.684562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:33.684562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:33.684562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:33.684562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:33.684562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:33.685169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:33.685169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:33.685169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:34.134446 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:34.134851 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:34.135097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:34.136391 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:34.136588 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:34.136719 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:34.136848 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:34.136968 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2579de9a-0ac0-40a2-a5d2-35c6997fae4c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:34.137103 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:34.137208 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:34.137337 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:34.137438 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:34.137541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:34.137637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:34.137717 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:34.137813 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-4687387f-d53d-4e48-9055-eaa429b9404d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:34.137913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.138411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:34.138411 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:34.138411 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:34.138411 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:34.138411 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:34.138411 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:34.138411 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.138411 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:34.139466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.139466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:34.139466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:34.139466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:34.139466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.139466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:34.139466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:34.139466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.139466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:34.139466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:34.139466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:34.139466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.139708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:34.139708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:34.139708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:34.139708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:34.139708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:34.139708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:34.139708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:34.139708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:34.139708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:34.139708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:34.139708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:34.139708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:34.139708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:34.139915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.139915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:34.140217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.140217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:34.140217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:34.140217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:34.140217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:34.140217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:34.140217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.140217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:34.140217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:34.140217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.140217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:34.140217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:34.140217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.140217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:34.140217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:34.140217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.141430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:34.141430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:34.141430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.141430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:34.141430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:34.141430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:34.141430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.141430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:34.141430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:34.141430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:34.141430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:34.141430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:34.141430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:34.141430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:34.141430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:34.141430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:34.141430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:34.141430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:34.141430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:34.142376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:34.142376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.142376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:34.156380 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:34.156823 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:34.156969 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:34.157183 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:25:34.157732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:34.158459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.158720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:34.158720 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:34.158720 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:34.158720 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:34.158720 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:34.158720 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:34.158720 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.158720 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:34.159595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.159595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:34.159595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:34.159595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:34.159595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.159595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:34.159595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:34.159595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:34.159595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.159595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:34.159595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:34.159595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:34.159595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:34.159595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:34.159595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:34.159908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:34.159908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:34.159908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:34.159908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:34.159908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:34.159908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:34.159908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:34.159908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.159908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:34.160800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.160800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:34.160800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:34.160800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:34.160800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.160800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:34.160800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:34.160800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:25:34.160800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:34.160800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:34.160800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.160800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:34.161178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:34.161178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:34.161178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.161178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:34.161178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:34.161178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:34.161178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:34.161178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:34.161178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:34.161178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:34.161178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:34.161178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:34.161178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:34.161178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:34.161444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:34.161444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:34.161444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.161444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:34.161444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.161444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:34.161444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:34.161444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:34.161444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:34.161444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:34.161444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.161444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:34.161444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:34.161444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.161705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:34.161705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:34.161705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.161705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:34.161705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:34.161705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.161705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:34.161705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:34.161705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:25:34.161705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:34.161705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:34.161705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.161705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:34.161705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:34.161705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:34.161705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.161705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:34.161705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:34.161705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:34.162109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:34.162109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:34.162109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:34.162109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:34.162109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:34.162109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:34.162109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:34.162109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:34.162109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:34.162109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:34.162109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.162109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:34.166702 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:34.167095 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:34.167242 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:34.167422 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:34.168723 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:34.168947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.170988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:34.170988 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:34.170988 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:34.170988 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:34.170988 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:34.170988 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:34.170988 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.170988 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:34.171239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.171239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:34.171239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:34.171239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:34.171239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.171239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:34.171239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:34.171239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:34.171239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.171239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:34.171239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:34.171239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:34.171239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:34.171239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:34.171239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:34.171515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:34.171515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:34.171515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:34.171515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:34.171515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:34.171515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:34.171515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:34.171515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.171515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:34.172741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.172741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:34.172741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:34.172741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:34.172741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.172741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:34.172741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:34.172741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:34.172741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:34.172741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.172741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:34.173136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:34.173136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:34.173136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.173136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:34.173136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:34.173136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:34.173136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:34.173136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:34.173136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:34.173136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:34.173136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:34.173136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:34.173136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:34.173136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:34.173495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:34.173495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:34.173495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.173495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:34.176309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.176309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:34.176309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:34.176309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:34.176309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:34.176309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:34.176309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.176309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:34.176309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:34.176309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.176309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:34.176309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:34.176309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.176309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:34.176309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:34.176309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.176309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:34.176962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:34.176962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:34.176962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:34.176962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.176962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:34.176962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:34.176962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:34.176962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.176962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:34.176962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:34.176962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:34.176962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:34.176962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:34.176962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:34.176962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:34.176962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:34.176962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:34.176962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:34.176962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:34.177503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:34.177503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:34.177503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.177503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:34.186599 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:34.187031 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:34.187231 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:34.187474 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:34.190548 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:34.190737 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:34.190999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.191484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:34.191484 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:34.191484 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:34.191484 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:34.191484 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:34.191484 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:34.191484 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.191484 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:34.192433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.192433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:34.192433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:34.192433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:34.192433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.192433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:34.192433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:34.192433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.192433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:34.192433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:34.192433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:34.192433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.192775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:34.192775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:34.192775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:34.192775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:34.192775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:34.192775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:34.192775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:34.192775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:34.192775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:34.192775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:34.192775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:34.192775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:34.192775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:34.193068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.193068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:34.193552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.193552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:34.193552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:34.193552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:34.193552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:34.193552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:34.193552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.193552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:34.193552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:34.193552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.193552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:34.193552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:34.193552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.193552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:34.193552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:34.193552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:34.193840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:34.193840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:34.193840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.193840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:34.193840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:34.193840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:34.193840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:34.193840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:34.193840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:34.193840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:34.193840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:34.193840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:34.193840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:34.193840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:34.193840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:34.193840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:34.193840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:34.193840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:34.193840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:34.194229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:34.194229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:34.194229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:38.617547 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:38.618622 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:38.618836 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:38.619301 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:38.681561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:38.681870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:38.686190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:38.686190 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:38.686190 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:38.686190 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:38.686190 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:38.686190 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:38.686190 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:38.686190 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:38.687132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.067 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:38.687132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:38.687132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:38.687132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:38.687132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:38.687132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:38.687132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:38.687132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:38.687132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:38.687132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:38.687132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:38.687132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:38.687562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:38.687562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:38.687562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:38.687562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:38.687562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:38.687562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:38.687562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:38.687562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:38.687562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:38.687562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:38.687562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:38.687562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:38.687562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:38.687805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:38.687805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:38.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:38.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:38.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:38.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:38.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:38.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:38.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:38.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:38.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:38.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:38.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:38.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:38.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:38.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:38.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:38.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:38.689004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:38.689004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:38.689004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:38.689004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:38.689004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:38.689004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:38.689004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:38.689004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:38.689004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:38.689004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:38.689004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:38.689004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:38.689004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:38.689004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:38.689004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:38.689004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:38.689004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:38.689004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:38.689004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:38.689558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:38.689558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:38.689558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:39.137722 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:39.138655 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:39.138788 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:39.141015 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:39.141259 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:39.142197 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:39.142368 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:39.142458 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:39.143369 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:39.143528 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:39.143628 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:39.143771 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:39.143863 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:39.143976 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:39.144116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.144876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:39.144876 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:39.144876 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:39.144876 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:39.144876 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:39.144876 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:39.144876 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.144876 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:39.147482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.147482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:39.147482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:39.147482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:39.147482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.147482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:39.147482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:39.147482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.147482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:39.147482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:39.147482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:39.147482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.148033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:39.148033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:39.148033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:39.148033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:39.148033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:39.148033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:39.148033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:39.148033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:39.148033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:39.148033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:39.148033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:39.148033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:39.148033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:39.148414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.148414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:39.149330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.149330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:39.149330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:39.149330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:39.149330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:39.149330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:39.149330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.149330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:39.149330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:39.149330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.149330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:39.149330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:39.149330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.149330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:39.149330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:39.149330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.149829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:39.149829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:39.149829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.149829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:39.149829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:39.149829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:39.149829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.149829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:39.149829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:39.149829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:39.149829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:39.149829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:39.149829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:39.149829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:39.149829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:39.149829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:39.149829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:39.149829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:39.149829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:39.151080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:39.151080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.151080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:39.156820 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:39.157857 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:39.157933 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:39.158128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:25:39.158691 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:39.158912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.159654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:39.159654 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:39.159654 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:39.159654 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:39.159654 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:39.159654 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:39.159654 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.159654 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:39.160418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.160418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:39.160418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:39.160418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:39.160418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.160418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:39.160418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:39.160418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:39.160418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.160418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:39.160418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:39.160418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:39.160418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:39.160418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:39.160418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:39.160731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:39.160731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:39.160731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:39.160731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:39.160731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:39.160731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:39.160731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:39.160731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.160731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:39.161218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.161218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:39.161218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:39.161218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:39.161218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.161218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:39.161218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:39.161218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:25:39.161218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:39.161218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:39.161218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.161218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:39.161573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:39.161573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:39.161573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.161573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:39.161573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:39.161573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:39.161573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:39.161573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:39.161573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:39.161573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:39.161573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:39.161573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:39.161573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:39.161573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:39.162001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:39.162001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:39.162001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.162001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:39.162563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.162563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:39.162563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:39.162563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:39.162563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:39.162563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:39.162563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.162563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:39.162563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:39.162563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.162563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:39.162563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:39.162563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.162563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:39.162563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:39.162563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.163364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:39.163364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:39.163364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:25:39.163364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:39.163364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:39.163364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.163364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:39.163364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:39.163364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:39.163364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.163364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:39.163364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:39.163364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:39.163364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:39.163364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:39.163364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:39.163364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:39.163364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:39.163960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:39.163960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:39.163960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:39.163960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:39.163960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:39.163960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.163960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:39.167146 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:39.168814 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:39.168937 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:39.169103 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:39.169471 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:39.169925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.172471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:39.172471 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:39.172471 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:39.172471 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:39.172471 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:39.172471 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:39.172471 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.172471 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:39.173244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.173244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:39.173244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:39.173244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:39.173244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.173244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:39.173244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:39.173244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:39.173244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.173244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:39.173244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:39.173244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:39.173244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:39.173244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:39.173244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:39.173699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:39.173699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:39.173699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:39.173699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:39.173699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:39.173699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:39.173699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:39.173699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.173699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:39.176541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.176541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:39.176541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:39.176541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:39.176541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.176541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:39.176541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:39.176541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:39.176541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:39.176541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.176541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:39.176958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:39.176958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:39.176958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.176958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:39.176958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:39.176958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:39.176958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:39.176958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:39.176958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:39.176958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:39.176958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:39.176958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:39.176958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:39.176958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:39.177947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:39.177947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:39.177947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.177947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:39.177947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.177947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:39.177947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:39.177947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:39.177947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:39.177947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:39.177947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.177947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:39.177947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:39.177947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.178437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:39.178437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:39.178437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.178437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:39.178437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:39.178437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.178437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:39.178437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:39.178437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:39.178437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:39.178437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.178437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:39.178437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:39.178437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:39.178437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.178437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:39.178437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:39.178437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:39.178437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:39.179471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:39.179471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:39.179471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:39.179471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:39.179471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:39.179471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:39.179471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:39.179471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:39.179471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:39.179471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.179471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:39.187022 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:39.187421 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:39.187612 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:39.187704 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:39.189308 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:39.189308 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:39.189308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.190123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:39.190123 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:39.190123 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:39.190123 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:39.190123 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:39.190123 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:39.190123 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.190123 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:39.190728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.190728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:39.190728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:39.190728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:39.190728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.190728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:39.190728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:39.190728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.190728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:39.190728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:39.190728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:39.190728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.191127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:39.191127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:39.191127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:39.191127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:39.191127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:39.191127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:39.191127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:39.191127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:39.191127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:39.191127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:39.191127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:39.191127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:39.191127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:39.191533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.191533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:39.192057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.192057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:39.192057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:39.192057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:39.192057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:39.192057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:39.192057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.192057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:39.192057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:39.192057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.192057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:39.192057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:39.192057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.192057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:39.192057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:39.192057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:39.192657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:39.192657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:39.192657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.192657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:39.192657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:39.192657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:39.192657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:39.192657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:39.192657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:39.192657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:39.192657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:39.192657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:39.192657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:39.192657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:39.192657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:39.192657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:39.192657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:39.192657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:39.192657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:39.193168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:39.193168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:39.193168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:43.619270 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:43.619703 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:43.619703 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:43.620508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:43.684135 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:43.684409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:43.685055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:43.685055 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:43.685055 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:43.685055 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:43.685055 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:43.685055 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:43.685055 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:43.685055 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:43.688535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.066 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:43.688535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:43.688535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:43.688535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:43.688535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:43.688535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:43.688535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:43.688535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:43.688535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:43.688535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:43.688535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:43.688535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:43.689016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:43.689016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:43.689016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:43.689016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:43.689016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:43.689016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:43.689016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:43.689016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:43.689016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:43.689016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:43.689016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:43.689016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:43.689016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:43.690147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:43.690147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:43.690147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:43.690147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:43.690147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:43.690147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:43.690147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:43.690147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:43.690147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:43.690147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:43.690147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:43.690147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:43.690147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:43.690147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:43.690147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:43.690869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:43.690869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:43.690869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:43.690869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:43.690869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:43.690869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:43.690869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:43.690869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:43.690869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:43.690869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:43.690869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:43.690869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:43.690869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:43.690869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:43.690869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:43.690869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:43.690869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:43.690869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:43.690869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:43.691426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:43.691426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:43.691426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:43.691426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:43.691426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:43.691426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:44.138727 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:44.139228 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:44.139345 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:44.143506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:44.143506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:44.143506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:44.143506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:44.143506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:44.144245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-76646331-a584-427a-b9a1-1fc67b87239c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:44.144245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:44.144245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:44.144245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:44.144245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:44.144482 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-0bfe0b73-2de6-448b-ac99-13c115d6ca54, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:44.144482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.144567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:44.144567 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:44.144567 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:44.144567 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:44.144567 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:44.144567 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:44.144567 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.144567 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:44.145670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.145670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:44.145670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:44.145670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:44.145670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.145670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:44.145670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:44.145670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.145670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:44.145670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:44.145670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:44.145670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.145888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:44.145888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:44.145888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:44.145888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:44.145888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:44.145888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:44.145888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:44.145888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:44.145888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:44.145888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:44.145888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:44.145888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:44.145888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:44.146154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.146154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:44.147557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.147557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:44.147557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:44.147557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:44.147557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:44.147557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:44.147557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.147557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:44.147557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:44.147557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.147557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:44.147557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:44.147557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.147557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:44.147557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:44.147557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.147926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:44.147926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:44.147926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.147926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:44.147926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:44.147926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:44.147926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.147926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:44.147926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:44.147926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:44.147926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:44.147926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:44.147926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:44.147926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:44.147926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:44.147926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:44.147926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:44.147926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:44.147926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:44.148426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:44.148426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.148426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:44.163193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:44.163193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:44.163193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:44.163193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:25:44.163193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:44.163193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.163553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:44.163553 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:44.163553 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:44.163553 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:44.163553 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:44.163553 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:44.163553 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.163553 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:44.163553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.163553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:44.163553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:44.163553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:44.163553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.163553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:44.163553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:44.163553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:44.163553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.163553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:44.164206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:44.164206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:44.164206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:44.164206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:44.164206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:44.164206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:44.164206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:44.164206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:44.164206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:44.164206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:44.164206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:44.164206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:44.164206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.164206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:44.164206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.164538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:44.164538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:44.164538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:44.164538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.164538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:44.164538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:44.164538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:25:44.164538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:44.164538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:44.164538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.164538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:44.164538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:44.164538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:44.165640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.165640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:44.165640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:44.165640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:44.165640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:44.165640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:44.165640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:44.165640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:44.165640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:44.165640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:44.165640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:44.165640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:44.165640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:44.165640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:44.166004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.166004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:44.166004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.166004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:44.166004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:44.166004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:44.166004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:44.166004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:44.166004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.166004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:44.166004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:44.166004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.166004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:44.166004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:44.166004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.167521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:44.167521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:44.167521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.167521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:44.167521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:44.167521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:25:44.167521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:44.167521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:44.167521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.167521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:44.167521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:44.167521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:44.167521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.167521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:44.167521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:44.167521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:44.167521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:44.167521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:44.167521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:44.168240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:44.168240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:44.168240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:44.168240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:44.168240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:44.168240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:44.168240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:44.168240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.168240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:44.169973 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:44.170762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:44.170876 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:44.171027 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:44.171374 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:44.171609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.172155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:44.172155 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:44.172155 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:44.172155 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:44.172155 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:44.172155 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:44.172155 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.172155 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:44.173093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.173093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:44.173093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:44.173093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:44.173093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.173093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:44.173093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:44.173093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:44.173093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.173093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:44.173093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:44.173093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:44.173093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:44.173093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:44.173093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:44.173534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:44.173534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:44.173534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:44.173534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:44.173534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:44.173534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:44.173534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:44.173534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.173534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:44.173890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.173890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:44.173890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:44.173890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:44.173890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.173890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:44.173890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:44.173890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:44.173890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:44.173890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.173890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:44.174218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:44.174218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:44.174218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.174218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:44.174218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:44.174218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:44.174218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:44.174218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:44.174218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:44.174218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:44.174218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:44.174218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:44.174218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:44.174218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:44.174594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:44.174594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:44.174594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.174594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:44.175092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.175092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:44.175092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:44.175092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:44.175092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:44.175092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:44.175092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.175092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:44.175092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:44.175092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.175092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:44.175092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:44.175092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.175092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:44.175092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:44.175092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.175092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:44.176345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:44.176345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:44.176345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:44.176345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.176345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:44.176345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:44.176345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:44.176345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.176345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:44.176345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:44.176345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:44.176345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:44.176345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:44.176345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:44.176345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:44.176345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:44.176345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:44.176345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:44.176345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:44.177334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:44.177334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:44.177334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.177334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:44.187780 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:44.188089 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:44.188216 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:44.188433 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:44.188776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:44.189009 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:44.189452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.190102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:44.190102 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:44.190102 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:44.190102 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:44.190102 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:44.190102 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:44.190102 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.190102 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:44.191344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.191344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:44.191344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:44.191344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:44.191344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.191344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:44.191344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:44.191344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.191344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:44.191344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:44.191344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:44.191344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.191707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:44.191707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:44.191707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:44.191707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:44.191707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:44.191707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:44.191707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:44.191707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:44.191707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:44.191707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:44.191707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:44.191707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:44.191707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:44.192034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.192034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:44.195635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.195635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:44.195635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:44.195635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:44.195635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:44.195635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:44.195635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.195635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:44.195635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:44.195635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.195635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:44.195635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:44.195635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.195635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:44.195635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:44.195635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:44.195969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:44.195969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:44.195969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.195969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:44.195969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:44.195969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:44.195969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:44.195969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:44.195969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:44.195969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:44.195969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:44.195969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:44.195969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:44.195969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:44.195969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:44.195969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:44.195969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:44.195969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:44.195969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:44.196311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:44.196311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:44.196311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:48.619337 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:48.622852 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:48.622973 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:48.623741 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:48.692614 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:48.692803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:48.693366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:48.693366 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:48.693366 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:48.693366 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:48.693366 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:48.693366 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:48.693366 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:48.693366 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:48.694082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.070 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:48.694082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:48.694082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:48.694082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:48.694082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:48.694082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:48.694082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:48.694082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:48.694082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:48.694082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:48.694082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:48.694082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:48.694508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:48.694508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:48.694508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:48.694508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:48.694508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:48.694508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:48.694508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:48.694508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:48.694508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:48.694508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:48.694508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:48.694508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:48.694508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:48.694868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:48.694868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:48.694868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:48.694868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:48.694868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:48.694868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:48.694868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:48.694868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:48.694868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:48.694868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:48.694868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:48.694868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:48.694868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:48.694868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:48.694868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:48.695323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:48.695323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:48.695323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:48.695323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:48.695323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:48.695323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:48.695323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:48.695323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:48.695323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:48.695323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:48.695323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:48.695323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:48.695323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:48.695323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:48.695323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:48.695323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:48.695323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:48.695323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:48.695323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:48.696578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:48.696578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:48.696578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:48.696578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:48.696578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:48.696578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:49.139320 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:49.142678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:49.142678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:49.143385 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:49.143614 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:49.143614 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:49.144773 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:49.144773 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:49.144773 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4bbcc5f6-c003-4c25-ac46-1d85e22f1e2a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:49.144773 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:49.144773 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:49.145008 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:49.145008 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-fcd9b94e-6481-4efa-9d12-3f30e261d2b9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:49.145114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.145639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:49.145639 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:49.145639 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:49.145639 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:49.145639 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:49.145639 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:49.145639 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.145639 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:49.146415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.146415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:49.146415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:49.146415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:49.146415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.146415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:49.146415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:49.146415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.146415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:49.146415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:49.146415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:49.146415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.146718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:49.146718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:49.146718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:49.146718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:49.146718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:49.146718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:49.146718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:49.146718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:49.146718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:49.146718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:49.146718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:49.146718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:49.146718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:49.146964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.146964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:49.147147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.147147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:49.147147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:49.147147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:49.147147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:49.147147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:49.147147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.147147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:49.147147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:49.147147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.147147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:49.147147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:49.147147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.147147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:49.147147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:49.147147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.147468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:49.147468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:49.147468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.147468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:49.147468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:49.147468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:49.147468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.147468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:49.147468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:49.147468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:49.147468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:49.147468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:49.147468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:49.147468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:49.147468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:49.147468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:49.147468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:49.147468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:49.147468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:49.147950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:49.147950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.147950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:49.159391 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:49.161707 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:49.161783 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:49.161899 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:25:49.162245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:49.162393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.164241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:49.164241 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:49.164241 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:49.164241 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:49.164241 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:49.164241 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:49.164241 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.164241 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:49.164782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.164782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:49.164782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:49.164782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:49.164782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.164782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:49.164782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:49.164782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:49.164782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.164782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:49.164782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:49.164782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:49.164782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:49.164782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:49.164782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:49.165193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:49.165193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:49.165193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:49.165193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:49.165193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:49.165193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:49.165193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:49.165193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.165193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:49.166294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.166294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:49.166294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:49.166294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:49.166294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.166294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:49.166294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:49.166294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:25:49.166294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:49.166294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:49.166294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.166294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:49.166580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:49.166580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:49.166580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.166580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:49.166580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:49.166580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:49.166580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:49.166580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:49.166580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:49.166580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:49.166580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:49.166580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:49.166580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:49.166580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:49.166863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:49.166863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:49.166863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.166863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:49.168342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.168342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:49.168342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:49.168342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:49.168342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:49.168342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:49.168342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.168342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:49.168342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:49.168342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.168342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:49.168342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:49.168342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.168342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:49.168342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:49.168342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.168676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:49.168676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:49.168676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:25:49.168676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:49.168676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:49.168676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.168676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:49.168676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:49.168676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:49.168676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.168676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:49.168676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:49.168676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:49.168676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:49.168676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:49.168676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:49.168676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:49.168676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:49.169013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:49.169013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:49.169013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:49.169013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:49.169013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:49.169013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.169013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:49.170812 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:49.171999 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:49.171999 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:49.171999 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:49.172290 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:49.172472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.173223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:49.173223 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:49.173223 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:49.173223 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:49.173223 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:49.173223 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:49.173223 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.173223 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:49.177436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.177436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:49.177436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:49.177436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:49.177436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.177436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:49.177436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:49.177436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:49.177436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.177436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:49.177436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:49.177436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:49.177436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:49.177436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:49.177436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:49.177773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:49.177773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:49.177773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:49.177773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:49.177773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:49.177773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:49.177773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:49.177773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.177773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:49.178185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.178185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:49.178185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:49.178185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:49.178185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.178185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:49.178185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:49.178185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:49.178185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:49.178185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.178185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:49.178711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:49.178711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:49.178711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.178711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:49.178711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:49.178711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:49.178711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:49.178711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:49.178711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:49.178711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:49.178711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:49.178711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:49.178711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:49.178711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:49.179012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:49.179012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:49.179012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.179012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:49.179012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.179012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:49.179012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:49.179012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:49.179012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:49.179012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:49.179012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.179012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:49.179012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:49.179012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.179360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:49.179360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:49.179360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.179360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:49.179360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:49.179360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.179360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:49.179360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:49.179360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:49.179360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:49.179360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.179360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:49.179360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:49.179360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:49.179360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.179360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:49.179360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:49.179360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:49.179360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:49.179707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:49.179707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:49.179707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:49.179707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:49.179707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:49.179707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:49.179707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:49.179707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:49.179707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:49.179707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.179707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:49.188215 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:49.188711 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:49.189385 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:49.189626 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:49.190101 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:49.190430 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:49.190790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.191409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:49.191409 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:49.191409 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:49.191409 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:49.191409 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:49.191409 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:49.191409 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.191409 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:49.192569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.192569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:49.192569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:49.192569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:49.192569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.192569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:49.192569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:49.192569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.192569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:49.192569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:49.192569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:49.192569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.192909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:49.192909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:49.192909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:49.192909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:49.192909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:49.192909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:49.192909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:49.192909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:49.192909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:49.192909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:49.192909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:49.192909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:49.192909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:49.193210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.193210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:49.193628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.193628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:49.193628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:49.193628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:49.193628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:49.193628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:49.193628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.193628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:49.193628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:49.193628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.193628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:49.193628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:49.193628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.193628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:49.193628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:49.193628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:49.193943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:49.193943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:49.193943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.193943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:49.193943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:49.193943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:49.193943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:49.193943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:49.193943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:49.193943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:49.193943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:49.193943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:49.193943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:49.193943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:49.193943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:49.193943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:49.193943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:49.193943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:49.193943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:49.194331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:49.194331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:49.194331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:53.622907 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:53.624204 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:53.625037 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:53.626327 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:53.630155 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=240 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:25:53.691329 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:53.691520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:53.692110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:53.692110 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:53.692110 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:53.692110 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:53.692110 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:53.692110 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:53.692110 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:53.692110 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:53.692784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.067 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:53.692784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:53.692784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:53.692784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:53.692784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:53.692784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:53.692784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:53.692784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:53.692784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:53.692784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:53.692784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:53.692784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:53.693089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:53.693089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:53.693089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:53.693089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:53.693089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:53.693089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:53.693089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:53.693089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:53.693089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:53.693089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:53.693089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:53.693089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:53.693089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:53.693391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:53.693391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:53.694575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:53.694575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:53.694575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:53.694575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:53.694575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:53.694575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:53.694575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:53.694575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:53.694575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:53.694575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:53.694575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:53.694575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:53.694575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:53.694575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:53.694575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:53.694575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:53.695269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:53.695269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:53.695269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:53.695269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:53.695269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:53.695269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:53.695269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:53.695269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:53.695269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:53.695269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:53.695269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:53.695269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:53.695269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:53.695269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:53.695269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:53.695269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:53.695269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:53.695269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:53.695269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:53.695637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:53.695637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:53.695637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:54.142469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:54.143803 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:54.144720 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:54.145934 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:54.146128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:54.146265 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:54.146402 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:54.146529 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:54.146735 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-17a1c9ab-f1ce-4bb8-8c91-34f88877fdba, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:54.146855 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-fcd9b94e-6481-4efa-9d12-3f30e261d2b9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:54.146984 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:54.147117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:54.147309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.147866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:54.147866 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:54.147866 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:54.147866 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:54.147866 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:54.147866 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:54.147866 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.147866 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:54.149066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.149066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:54.149066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:54.149066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:54.149066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.149066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:54.149066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:54.149066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.149066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:54.149066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:54.149066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:54.149066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.149557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:54.149557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:54.149557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:54.149557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:54.149557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:54.149557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:54.149557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:54.149557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:54.149557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:54.149557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:54.149557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:54.149557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:54.149557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:54.150029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.150029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:54.150029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.150029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:54.150029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:54.150029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:54.150029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:54.150029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:54.150029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.150029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:54.150029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:54.150029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.150029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:54.150029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:54.150029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.150794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:54.150794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:54.150794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.150794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:25:54.150794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:54.150794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.150794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:54.150794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:54.150794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:54.150794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.150794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:54.150794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:54.150794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:54.150794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:54.150794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:54.150794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:54.150794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:54.150794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:54.150794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:54.151471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:54.151471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:54.151471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:54.151471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:54.151471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.151471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:54.161762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:54.162126 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:54.162221 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:54.162411 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:25:54.164309 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:54.164515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.165041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:54.165041 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:54.165041 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:54.165041 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:54.165041 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:54.165041 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:54.165041 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.165041 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:54.165714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.165714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:54.165714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:54.165714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:54.165714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.165714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:54.165714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:54.165714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:54.165714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.165714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:54.165714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:54.165714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:54.165714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:54.165714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:54.165714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:54.166028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:54.166028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:54.166028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:54.166028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:54.166028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:54.166028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:54.166028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:54.166028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.166028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:54.166464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.166464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:54.166464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:54.166464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:54.166464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.166464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:54.166464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:54.166464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:25:54.166464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:54.166464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:54.166464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.166464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:54.166731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:54.166731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:54.166731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.166731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:54.166731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:54.166731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:54.166731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:54.166731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:54.166731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:54.166731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:54.166731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:54.166731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:54.166731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:54.166731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:54.166974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:54.166974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:54.166974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.166974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:54.167068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.167068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:54.167068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:54.167068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:54.167068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:54.167068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:54.167068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.167068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:54.167068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:54.167068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.167068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:54.167068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:54.167068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.167068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:54.167068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:54.167068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.167397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:25:54.167397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:25:54.167397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:25:54.167397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:54.167397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:54.167397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.167397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:54.167397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:54.167397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:54.167397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.167397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:54.167397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:54.167397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:54.167397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:54.167397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:54.167397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:54.167397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:54.167397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:54.167715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:54.167715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:54.167715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:54.167715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:54.167715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:54.167715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.167715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:54.172425 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:54.174458 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:54.174590 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:54.174925 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:54.176181 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:54.176457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.177010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:54.177010 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:54.177010 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:54.177010 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:54.177010 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:54.177010 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:54.177010 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.177010 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:54.178329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.178329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:25:54.178329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:54.178329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:54.178329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.178329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:54.178329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:54.178329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:25:54.178329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.178329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:25:54.178329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:54.178329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:25:54.178329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:54.178329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:25:54.178329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:25:54.178593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:54.178593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:25:54.178593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:54.178593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:25:54.178593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:25:54.178593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:54.178593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:25:54.178593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.178593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:25:54.179807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.179807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:54.179807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:54.179807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:54.179807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.179807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:54.179807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:54.179807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:54.179807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:54.179807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.179807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:54.180044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:54.180044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:54.180044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.180044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:54.180044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:54.180044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:54.180044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:54.180044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:54.180044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:54.180044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:54.180044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:54.180044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:54.180044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:54.180044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:54.180429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:54.180429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:54.180429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.180429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:54.181533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.181533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:54.181533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:54.181533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:54.181533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:54.181533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:54.181533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.181533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:54.181533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:54.181533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.181533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:54.181533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:54.181533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.181533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:54.181533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:54.181533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.181533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:25:54.181858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:25:54.181858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:25:54.181858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:25:54.181858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.181858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:54.181858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:54.181858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:54.181858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.181858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:54.181858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:54.181858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:54.181858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:54.181858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:54.181858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:54.181858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:54.181858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:54.181858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:54.181858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:54.181858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:54.182156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:54.182156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:54.182156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.182156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:54.188805 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:54.189238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:54.189703 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:54.189924 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:54.190891 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:54.191143 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:54.191508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.191986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:54.191986 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:54.191986 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:54.191986 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:54.191986 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:54.191986 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:54.191986 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.191986 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:54.193076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.193076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:54.193076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:54.193076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:54.193076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.193076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:54.193076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:54.193076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.193076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:54.193076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:54.193076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:54.193076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.193503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:54.193503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:54.193503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:54.193503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:54.193503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:54.193503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:54.193503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:54.193503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:54.193503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:54.193503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:54.193503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:54.193503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:54.193503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:54.193954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.193954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:54.194359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.194359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:54.194359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:54.194359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:54.194359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:54.194359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:54.194359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.194359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:54.194359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:54.194359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.194359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:54.194359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:54.194359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.194359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:54.194359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:54.194359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:54.194838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:25:54.194838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:54.194838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.194838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:54.194838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:54.194838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:54.194838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:54.194838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:54.194838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:54.194838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:54.194838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:54.194838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:54.194838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:54.194838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:54.194838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:54.194838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:54.194838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:54.194838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:54.194838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:54.195663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:54.195663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:54.195663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:58.625389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:58.626131 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:58.626452 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:58.627191 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:25:58.683559 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:58.683829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.684457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:58.684457 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:58.684457 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:58.684457 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:58.684457 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:58.684457 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:58.684457 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.684457 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:58.685575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.685575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:58.685575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:58.685575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:25:58.685575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:58.685575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:58.685575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:58.685575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:58.685575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:58.685575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:58.685575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:58.685575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:58.685844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:58.685844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:58.685844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:58.685844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:58.685844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:58.685844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:58.685844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:58.685844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:58.685844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:58.685844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:58.685844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:58.685844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:58.685844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:58.686085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.686085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:58.686576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.686576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:25:58.686576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:25:58.686576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:25:58.686576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:25:58.686576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:25:58.686576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:58.686576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:25:58.686576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:25:58.686576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:58.686576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:25:58.686576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:25:58.686576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:25:58.686576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:25:58.686576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:25:58.686576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:25:58.686907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:25:58.686907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:58.686907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:58.686907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:58.686907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:58.686907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:25:58.686907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:58.686907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:25:58.686907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:58.686907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:25:58.686907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:58.686907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:25:58.686907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:25:58.686907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:58.686907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:25:58.686907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:58.686907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:25:58.686907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:25:58.686907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:58.687323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:25:58.687323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.687323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:25:58.857661 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:58.858116 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:25:58.858116 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:25:58.868729 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 08 14:25:58.868985 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Number of inconsistencies found at create/update: networks=12, security_groups=1, routers=6, router_ports=6 {{(pid=89274) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 14:25:58.869137 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=10 {{(pid=89274) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 14:25:58.869282 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource a3a109cb-260d-4fcb-a3f7-5844e08a48f8 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:25:58.899966 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-a3a109cb-260d-4fcb-a3f7-5844e08a48f8, resource={'id': 'a3a109cb-260d-4fcb-a3f7-5844e08a48f8', 'name': 'tempest-TaggedBootDevicesTest_v242-1121652030-network', 'tenant_id': '2b04558547c940a996e4b07f1a6736e5', 'project_id': '2b04558547c940a996e4b07f1a6736e5', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['c2b141da-f159-4dee-89c9-47663a255daa'], 'standard_attr_id': 136, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:23:36Z', 'updated_at': '2026-06-08T14:23:37Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 42494}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:58.900124 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-a3a109cb-260d-4fcb-a3f7-5844e08a48f8, col_values=(('external_ids', {'neutron:network_name': 'tempest-TaggedBootDevicesTest_v242-1121652030-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:58.900291 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-68b86895-5e21-4994-8031-dfb52a1765ce, external_ids_update=None, columns={'tag': 42494}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:58.900381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.900802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:58.900802 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:58.900802 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:58.900802 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:58.900802 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:58.900802 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:58.900802 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.900802 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:58.902154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource a3a109cb-260d-4fcb-a3f7-5844e08a48f8 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.902154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:58.902154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:25:58.902154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:25:58.902154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:25:58.902154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:25:58.902154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:25:58.902154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:58.902154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:58.902154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:58.902154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:58.902618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:58.902618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:58.902618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:58.902618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:58.902618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:58.902618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:58.902618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:58.902618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:58.902618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:58.902618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:58.902618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:58.902618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:58.902618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:58.902618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:58.902885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:58.902885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.902885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:58.902885 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 673fa88a-f8fc-4333-941d-9e969d3d75a3 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:25:58.925877 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-673fa88a-f8fc-4333-941d-9e969d3d75a3, resource={'id': '673fa88a-f8fc-4333-941d-9e969d3d75a3', 'name': 'tempest-device-tagging-net2-1017538534', 'tenant_id': '14748d9641a743a8bc8187ab6ae4b82b', 'project_id': '14748d9641a743a8bc8187ab6ae4b82b', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['7410d108-8c13-4aaa-9ed2-a64e573f28d2'], 'standard_attr_id': 257, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:24:02Z', 'updated_at': '2026-06-08T14:24:04Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 48693}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:58.926085 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-673fa88a-f8fc-4333-941d-9e969d3d75a3, col_values=(('external_ids', {'neutron:network_name': 'tempest-device-tagging-net2-1017538534', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:58.926302 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-47abfc08-7335-4056-a540-4d64d68a75e1, external_ids_update=None, columns={'tag': 48693}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:58.926416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.926833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:58.926833 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:58.926833 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:58.926833 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:58.926833 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:58.926833 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:58.926833 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.926833 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:58.927607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 673fa88a-f8fc-4333-941d-9e969d3d75a3 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.927607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:58.927607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:25:58.927607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:25:58.927607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:25:58.927607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:25:58.927607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:25:58.927607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:58.927607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:58.927607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:58.927607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:58.927846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:58.927846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:58.927846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:58.927846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:58.927846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:58.927846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:58.927846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:58.927846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:58.927846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:58.927846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:58.927846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:58.927846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:58.927846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:58.927846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:58.928154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:58.928154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.928154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:58.928154 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource b6de457d-091a-439c-8caf-d63384a74d06 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:25:58.954850 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-b6de457d-091a-439c-8caf-d63384a74d06, resource={'id': 'b6de457d-091a-439c-8caf-d63384a74d06', 'name': 'tempest-device-tagging-net1-1109816481', 'tenant_id': '14748d9641a743a8bc8187ab6ae4b82b', 'project_id': '14748d9641a743a8bc8187ab6ae4b82b', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['f5da607e-3b77-4584-86a5-b1da5daba830'], 'standard_attr_id': 253, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:24:01Z', 'updated_at': '2026-06-08T14:24:03Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 36867}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:58.955174 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-b6de457d-091a-439c-8caf-d63384a74d06, col_values=(('external_ids', {'neutron:network_name': 'tempest-device-tagging-net1-1109816481', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:58.955568 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-61acdb98-9626-45b3-8e30-3ae826058740, external_ids_update=None, columns={'tag': 36867}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:58.955729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.956404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:58.956404 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:58.956404 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:58.956404 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:58.956404 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:58.956404 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:58.956404 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.956404 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:58.957419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource b6de457d-091a-439c-8caf-d63384a74d06 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.957419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:58.957419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:25:58.957419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:25:58.957419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:25:58.957419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:25:58.957419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:25:58.957419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:58.957419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:58.957419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:58.957419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:58.957646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:58.957646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:58.957646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:58.957646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:58.957646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:58.957646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:58.957646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:58.957646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:58.957646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:58.957646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:58.957646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:58.957646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:58.957646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:58.957646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:58.957959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:58.957959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.957959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:58.957959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 0f60932f-6715-4655-866a-5ad5156eebf1 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:25:58.993784 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-0f60932f-6715-4655-866a-5ad5156eebf1, resource={'id': '0f60932f-6715-4655-866a-5ad5156eebf1', 'name': 'tempest-tagged-attachments-test-net-553458358', 'tenant_id': '143396ea48cb4e8a9aef7b2da4ae9da8', 'project_id': '143396ea48cb4e8a9aef7b2da4ae9da8', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['01a6bda4-51e4-419d-b44f-efeb67182e09'], 'standard_attr_id': 212, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:23:51Z', 'updated_at': '2026-06-08T14:23:53Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 55092}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:58.994740 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-0f60932f-6715-4655-866a-5ad5156eebf1, col_values=(('external_ids', {'neutron:network_name': 'tempest-tagged-attachments-test-net-553458358', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:58.996589 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-ffb615e3-8de4-48b6-be87-7d9b0855d59c, external_ids_update=None, columns={'tag': 55092}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:58.996948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.997573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:58.997573 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:58.997573 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:58.997573 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:58.997573 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:58.997573 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:58.997573 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.997573 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:58.998502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 0f60932f-6715-4655-866a-5ad5156eebf1 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.998502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:58.998502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:25:58.998502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:25:58.998502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:25:58.998502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:25:58.998502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:25:58.998502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:58.998502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:58.998502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:58.998502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:58.998832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:58.998832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:58.998832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:58.998832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:58.998832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:58.998832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:58.998832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:58.998832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:58.998832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:58.998832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:58.998832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:58.998832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:58.998832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:58.998832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:58.999245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:58.999245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:58.999245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:58.999245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 7043cb42-9bff-47f8-9ecd-2b25c2ca7e58 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:25:59.035689 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-7043cb42-9bff-47f8-9ecd-2b25c2ca7e58, resource={'id': '7043cb42-9bff-47f8-9ecd-2b25c2ca7e58', 'name': 'tempest-TaggedAttachmentsTest-1722623665-network', 'tenant_id': '2720169bfd6b44daaa1ea0a963fba116', 'project_id': '2720169bfd6b44daaa1ea0a963fba116', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['42c0c945-19d8-46b4-ae5f-c5c5cfb0aeef'], 'standard_attr_id': 169, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:23:38Z', 'updated_at': '2026-06-08T14:23:39Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 23419}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.035917 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-7043cb42-9bff-47f8-9ecd-2b25c2ca7e58, col_values=(('external_ids', {'neutron:network_name': 'tempest-TaggedAttachmentsTest-1722623665-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.036163 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-562bb23d-606c-4d08-ad5f-d05164d6156e, external_ids_update=None, columns={'tag': 23419}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.036283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.036869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:59.036869 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.036869 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:59.036869 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:59.036869 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.036869 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:59.036869 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.036869 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:59.037827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 7043cb42-9bff-47f8-9ecd-2b25c2ca7e58 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.037827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:59.037827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:25:59.037827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:25:59.037827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:25:59.037827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:25:59.037827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:25:59.037827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:59.037827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.037827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.037827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:59.038308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:59.038308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.038308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.038308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:59.038308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:59.038308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:59.038308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:59.038308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:59.038308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:59.038308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:59.038308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.038308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:59.038308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:59.038308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.038733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:59.038733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.038733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:59.038733 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 48166942-dd74-4a69-80ee-737e56e5cff4 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:25:59.071099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-48166942-dd74-4a69-80ee-737e56e5cff4, resource={'id': '48166942-dd74-4a69-80ee-737e56e5cff4', 'name': 'tempest-TaggedBootDevicesTest-1814720201-network', 'tenant_id': '8ecbd34f4df34a9f8209f149eab98486', 'project_id': '8ecbd34f4df34a9f8209f149eab98486', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['d3662c83-5220-46e8-a89f-17fbfd64ce14'], 'standard_attr_id': 160, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:23:37Z', 'updated_at': '2026-06-08T14:23:39Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 21559}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.071099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-48166942-dd74-4a69-80ee-737e56e5cff4, col_values=(('external_ids', {'neutron:network_name': 'tempest-TaggedBootDevicesTest-1814720201-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.071806 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-e05cda53-0e78-455e-916f-b0c54a8579ca, external_ids_update=None, columns={'tag': 21559}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.071806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.072213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:59.072213 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.072213 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:59.072213 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:59.072213 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.072213 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:59.072213 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.072213 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:59.073336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 48166942-dd74-4a69-80ee-737e56e5cff4 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.073336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:59.073336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:25:59.073336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:25:59.073336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:25:59.073336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:25:59.073336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:25:59.073336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:59.073336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.073336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.073336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:59.073626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:59.073626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.073626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.073626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:59.073626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:59.073626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:59.073626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:59.073626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:59.073626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:59.073626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:59.073626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.073626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:59.073626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:59.073626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.073925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:59.073925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.073925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:59.073925 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource a137566a-eaf1-4f5d-a192-15d63e6c049a (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:25:59.123678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-a137566a-eaf1-4f5d-a192-15d63e6c049a, resource={'id': 'a137566a-eaf1-4f5d-a192-15d63e6c049a', 'name': 'private', 'tenant_id': 'ba473288446843d3a29e25267968ba61', 'project_id': 'ba473288446843d3a29e25267968ba61', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['7b3cb50f-b942-4054-b735-5f7be989b8f8', 'ba10f097-6c2b-4188-8d42-e9415eff0039'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:20:42Z', 'updated_at': '2026-06-08T14:20:44Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 26551}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.123921 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-a137566a-eaf1-4f5d-a192-15d63e6c049a, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.124222 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-c4b70b51-9d46-4e6b-a1e7-b3e6a6b8b666, external_ids_update=None, columns={'tag': 26551}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.124356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.124891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:59.124891 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.124891 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:59.124891 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:59.124891 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.124891 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:59.124891 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.124891 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:59.125762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource a137566a-eaf1-4f5d-a192-15d63e6c049a (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.125762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:59.125762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:25:59.125762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:25:59.125762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:25:59.125762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:25:59.125762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:25:59.125762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:59.125762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.125762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.125762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:59.126175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:59.126175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.126175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.126175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:59.126175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:59.126175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:59.126175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:59.126175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:59.126175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:59.126175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:59.126175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.126175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:59.126175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:59.126175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.126615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:59.126615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.126615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:59.126615 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource fffd4e9d-b343-4a97-8e5f-4015f805fc46 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:25:59.145741 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:59.163169 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:59.163596 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, resource={'id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'name': 'public', 'tenant_id': 'ba473288446843d3a29e25267968ba61', 'project_id': 'ba473288446843d3a29e25267968ba61', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['4ee83802-2181-432f-a76d-d3335bbf4e0b', 'b9859e49-8026-492d-a0a4-11326477713e'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-08T14:20:46Z', 'updated_at': '2026-06-08T14:20:52Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.163740 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.164011 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-0817ce61-1c85-4457-911a-0b8fdccf389d, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.164205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.164673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:59.164673 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.164673 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:59.164673 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:59.164673 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.164673 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:59.164673 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.164673 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:59.168688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource fffd4e9d-b343-4a97-8e5f-4015f805fc46 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.168688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:59.168688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:25:59.168688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:25:59.168688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:25:59.168688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:25:59.168688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:25:59.168688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:59.168688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.168688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.168688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:59.169197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:59.169197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.169197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.169197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:59.169197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:59.169197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:59.169197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:59.169197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:59.169197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:59.169197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:59.169197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.169197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:59.169197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:59.169197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.170213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:59.170213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.170213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:59.170213 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 73426422-a7fc-44cd-9fc5-cd2d8fa95c38 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:25:59.175977 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:59.191676 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:25:59.222497 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-73426422-a7fc-44cd-9fc5-cd2d8fa95c38, resource={'id': '73426422-a7fc-44cd-9fc5-cd2d8fa95c38', 'name': 'shared', 'tenant_id': 'ba473288446843d3a29e25267968ba61', 'project_id': 'ba473288446843d3a29e25267968ba61', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['2b53474a-74e5-4c44-bb1b-86c8215bd7ab'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:21:52Z', 'updated_at': '2026-06-08T14:21:52Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 292}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.222934 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-73426422-a7fc-44cd-9fc5-cd2d8fa95c38, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.223358 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-5c6a587e-b2f4-42c7-9706-8a2e20b0df46, external_ids_update=None, columns={'tag': 292}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.223834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.224611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:59.224611 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.224611 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:59.224611 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:59.224611 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.224611 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:59.224611 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.224611 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:59.228017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 73426422-a7fc-44cd-9fc5-cd2d8fa95c38 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.228017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:59.228017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:25:59.228017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:25:59.228017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:25:59.228017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:25:59.228017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:25:59.228017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:59.228017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.228017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.228017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:59.228367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:59.228367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.228367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.228367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:59.228367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:59.228367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:59.228367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:59.228367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:59.228367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:59.228367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:59.228367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.228367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:59.228367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:59.228367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.228796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:59.228796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.228796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:59.228796 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 8422f4e0-ae6e-4fee-a586-3260d9cd63b6 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:25:59.268266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-8422f4e0-ae6e-4fee-a586-3260d9cd63b6, resource={'id': '8422f4e0-ae6e-4fee-a586-3260d9cd63b6', 'name': 'tempest-ServerBootFromVolumeStableRescueTest-1010071599-network', 'tenant_id': '2894d8d8f05c4913a22043e6761c5718', 'project_id': '2894d8d8f05c4913a22043e6761c5718', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['8c6f3d78-ebcd-4cf8-bc10-3e98d2780e92'], 'standard_attr_id': 40, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:23:17Z', 'updated_at': '2026-06-08T14:23:19Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 35332}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.268266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-8422f4e0-ae6e-4fee-a586-3260d9cd63b6, col_values=(('external_ids', {'neutron:network_name': 'tempest-ServerBootFromVolumeStableRescueTest-1010071599-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.268463 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-2fb0edf8-0391-4a7b-beb1-ffb1a09cd2cb, external_ids_update=None, columns={'tag': 35332}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.268463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.268463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:59.268463 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.268463 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:59.268463 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:59.268463 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.268463 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:59.268463 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.268463 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:59.269241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 8422f4e0-ae6e-4fee-a586-3260d9cd63b6 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.269241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:59.269241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:25:59.269241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:25:59.269241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:25:59.269241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:25:59.269241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:25:59.269241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:59.269241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.269241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.269241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:59.269523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:59.269523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.269523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.269523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:59.269523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:59.269523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:59.269523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:59.269523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:59.269523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:59.269523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:59.269523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.269523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:59.269523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:59.269523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.269773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:59.269773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.269773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:59.269773 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource c1eb84a3-0388-4f56-846f-a4476ce83ed2 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:25:59.301981 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-c1eb84a3-0388-4f56-846f-a4476ce83ed2, resource={'id': 'c1eb84a3-0388-4f56-846f-a4476ce83ed2', 'name': 'tempest-TaggedAttachmentsTest-1758790243-network', 'tenant_id': '143396ea48cb4e8a9aef7b2da4ae9da8', 'project_id': '143396ea48cb4e8a9aef7b2da4ae9da8', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['f540a72a-894e-49b1-9e56-5b5bfeafb262'], 'standard_attr_id': 88, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:23:19Z', 'updated_at': '2026-06-08T14:23:21Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 48064}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.303872 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-c1eb84a3-0388-4f56-846f-a4476ce83ed2, col_values=(('external_ids', {'neutron:network_name': 'tempest-TaggedAttachmentsTest-1758790243-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.304495 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-2bb410c9-ab41-4466-917f-e09b8a043452, external_ids_update=None, columns={'tag': 48064}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.304732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.305423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:59.305423 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.305423 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:59.305423 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:59.305423 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.305423 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:59.305423 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.305423 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:59.306386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource c1eb84a3-0388-4f56-846f-a4476ce83ed2 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.306386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:59.306386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:25:59.306386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:25:59.306386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:25:59.306386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:25:59.306386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:25:59.306386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:59.306386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.306386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.306386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:59.306795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:59.306795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.306795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.306795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:59.306795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:59.306795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:59.306795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:59.306795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:59.306795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:59.306795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:59.306795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.306795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:59.306795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:59.306795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.307239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:59.307239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.307239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:59.307239 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource a434afde-d1cc-430f-a938-fa85941e1ace (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:25:59.342774 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-a434afde-d1cc-430f-a938-fa85941e1ace, resource={'id': 'a434afde-d1cc-430f-a938-fa85941e1ace', 'name': 'tempest-TaggedBootDevicesTest-235969666-network', 'tenant_id': '14748d9641a743a8bc8187ab6ae4b82b', 'project_id': '14748d9641a743a8bc8187ab6ae4b82b', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['4b377a0e-ca82-4cb9-a709-cf01748b3c69'], 'standard_attr_id': 79, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:23:19Z', 'updated_at': '2026-06-08T14:23:20Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 25996}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.342962 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-a434afde-d1cc-430f-a938-fa85941e1ace, col_values=(('external_ids', {'neutron:network_name': 'tempest-TaggedBootDevicesTest-235969666-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.343201 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-41cacc7a-2ab6-4b41-a8fa-c30430dc21cf, external_ids_update=None, columns={'tag': 25996}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.344746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.345495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:59.345495 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.345495 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:59.345495 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:59.345495 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.345495 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:59.345495 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.345495 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:59.347290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource a434afde-d1cc-430f-a938-fa85941e1ace (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.347290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:59.347290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:25:59.347290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:25:59.347290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:25:59.347290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:25:59.347290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:25:59.347290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:59.347290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.347290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.347290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:59.347619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:59.347619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.347619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.347619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:59.347619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:59.347619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:59.347619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:59.347619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:59.347619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:59.347619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:59.347619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.347619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:59.347619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:59.347619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.347912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:59.347912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.347912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:59.347912 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource b7df3be2-880e-43c0-bc8f-9c52302b1116 (type: security_groups) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:25:59.389869 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Successfully bumped revision number for resource b7df3be2-880e-43c0-bc8f-9c52302b1116 (type: security_groups) to 3 Jun 08 14:25:59.390026 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 4a3c780d-da5d-48d5-8bb6-d81ad7f30477 (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:25:59.481073 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, resource={'id': '4a3c780d-da5d-48d5-8bb6-d81ad7f30477', 'name': 'tempest-TaggedBootDevicesTest-235969666-router', 'project_id': '14748d9641a743a8bc8187ab6ae4b82b', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.128'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::24e'}], 'enable_snat': True}, 'gw_port_id': '7e30389c-1ed3-4581-91b8-fe46c4ae387f', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.128', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::24e', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:23:24Z', 'updated_at': '2026-06-08T14:23:34Z', 'revision_number': 3, 'tenant_id': '14748d9641a743a8bc8187ab6ae4b82b'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.481333 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, columns={'external_ids': {'neutron:router_name': 'tempest-TaggedBootDevicesTest-235969666-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.481508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "7e30389c-1ed3-4581-91b8-fe46c4ae387f"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.481622 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "7e30389c-1ed3-4581-91b8-fe46c4ae387f"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.482518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.482518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:59.482518 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.482518 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:59.482518 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:59.482518 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.482518 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:59.482518 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.482518 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:59.482878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router 4a3c780d-da5d-48d5-8bb6-d81ad7f30477. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.484085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 4a3c780d-da5d-48d5-8bb6-d81ad7f30477 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.484085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:59.484085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:25:59.484085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:25:59.484085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:25:59.484085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:25:59.484085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:25:59.484085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:25:59.484085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:25:59.484085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:25:59.484085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:25:59.484499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:25:59.484499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:25:59.484499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:59.484499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.484499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.484499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:59.484499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:59.484499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.484499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.484499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:59.484499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:59.484499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:59.484499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:59.484499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:59.485010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:59.485010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:59.485010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.485010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:59.485010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:59.485010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.485010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:59.485010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.485010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:59.485010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 64129b0c-35f5-448b-874d-cdf90527f8fd (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:25:59.551619 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, resource={'id': '64129b0c-35f5-448b-874d-cdf90527f8fd', 'name': 'tempest-TaggedAttachmentsTest-1722623665-router', 'project_id': '2720169bfd6b44daaa1ea0a963fba116', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.36'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::1ae'}], 'enable_snat': True}, 'gw_port_id': '37b1707e-d3f2-4888-8a3f-ee37f8f220a8', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.36', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::1ae', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:23:42Z', 'updated_at': '2026-06-08T14:23:49Z', 'revision_number': 3, 'tenant_id': '2720169bfd6b44daaa1ea0a963fba116'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.551782 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, columns={'external_ids': {'neutron:router_name': 'tempest-TaggedAttachmentsTest-1722623665-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.551957 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "37b1707e-d3f2-4888-8a3f-ee37f8f220a8"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.552073 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "37b1707e-d3f2-4888-8a3f-ee37f8f220a8"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.552163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.552599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:59.552599 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.552599 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:59.552599 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:59.552599 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.552599 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:59.552599 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.552599 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:59.552845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router 64129b0c-35f5-448b-874d-cdf90527f8fd. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.553665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 64129b0c-35f5-448b-874d-cdf90527f8fd (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.553665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:59.553665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:25:59.553665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:25:59.553665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:25:59.553665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:25:59.553665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:25:59.553665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:25:59.553665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:25:59.553665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:25:59.553665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:25:59.554049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:25:59.554049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:25:59.554049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:59.554049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.554049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.554049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:59.554049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:59.554049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.554049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.554049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:59.554049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:59.554049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:59.554049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:59.554049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:59.554518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:59.554518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:59.554518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.554518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:59.554518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:59.554518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.554518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:59.554518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.554518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:59.554518 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 9c10de0a-4866-44f0-a1a0-4c1a9070072d (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:25:59.624818 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, resource={'id': '9c10de0a-4866-44f0-a1a0-4c1a9070072d', 'name': 'tempest-TaggedBootDevicesTest-1814720201-router', 'project_id': '8ecbd34f4df34a9f8209f149eab98486', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.166'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::106'}], 'enable_snat': True}, 'gw_port_id': '98f87ce8-1f5e-4abd-b6de-32f21dbb1d4b', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.166', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::106', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:23:41Z', 'updated_at': '2026-06-08T14:23:47Z', 'revision_number': 3, 'tenant_id': '8ecbd34f4df34a9f8209f149eab98486'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.625077 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, columns={'external_ids': {'neutron:router_name': 'tempest-TaggedBootDevicesTest-1814720201-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.625248 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "98f87ce8-1f5e-4abd-b6de-32f21dbb1d4b"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.625393 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "98f87ce8-1f5e-4abd-b6de-32f21dbb1d4b"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.626134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.626134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:59.626134 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.626134 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:59.626134 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:59.626134 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.626134 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:59.626134 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.626134 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:59.626398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router 9c10de0a-4866-44f0-a1a0-4c1a9070072d. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.627568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 9c10de0a-4866-44f0-a1a0-4c1a9070072d (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.627568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:59.627568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:25:59.627568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:25:59.627568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:25:59.627568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:25:59.627568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:25:59.627568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:25:59.627568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:25:59.627568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:25:59.627568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:25:59.627817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:25:59.627817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:25:59.627817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:59.627817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.627817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.627817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:59.627817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:59.627817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.627817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.627817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:59.627817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:59.627817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:59.627817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:59.627817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:59.628192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:59.628192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:59.628192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.628192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:59.628192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:59.628192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.628192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:59.628192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.628192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:59.628192 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 17e5b0cf-4ee1-4902-9b75-92c18f37aafd (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:25:59.699824 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, resource={'id': '17e5b0cf-4ee1-4902-9b75-92c18f37aafd', 'name': 'tempest-TaggedBootDevicesTest_v242-1121652030-router', 'project_id': '2b04558547c940a996e4b07f1a6736e5', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.129'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::329'}], 'enable_snat': True}, 'gw_port_id': '49bacaa1-8ede-4596-ba0a-64ecc3a9a217', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.129', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::329', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:23:39Z', 'updated_at': '2026-06-08T14:23:43Z', 'revision_number': 3, 'tenant_id': '2b04558547c940a996e4b07f1a6736e5'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.706913 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, columns={'external_ids': {'neutron:router_name': 'tempest-TaggedBootDevicesTest_v242-1121652030-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.706913 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "49bacaa1-8ede-4596-ba0a-64ecc3a9a217"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.706913 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "49bacaa1-8ede-4596-ba0a-64ecc3a9a217"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.706913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.706913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:59.706913 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.706913 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:59.706913 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:59.707231 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.707231 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:59.707231 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.707231 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:59.707231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router 17e5b0cf-4ee1-4902-9b75-92c18f37aafd. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.707231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 17e5b0cf-4ee1-4902-9b75-92c18f37aafd (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.707231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:59.707231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:25:59.707231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:25:59.707231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:25:59.707231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:25:59.707627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:25:59.707627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:25:59.707627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:25:59.707627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:25:59.707627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:25:59.707627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:25:59.707627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:25:59.707627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:59.707627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.707627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.707627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:59.707627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:59.707627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.707947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.707947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:59.707947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:59.707947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:59.707947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:59.707947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:59.707947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:59.707947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:59.707947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.707947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:59.707947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:59.707947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.707947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:59.708634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.708634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:59.708634 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 55f4749b-3181-4ab1-8c76-c41325e4f742 (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:25:59.762148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, resource={'id': '55f4749b-3181-4ab1-8c76-c41325e4f742', 'name': 'tempest-ServerBootFromVolumeStableRescueTest-1010071599-router', 'project_id': '2894d8d8f05c4913a22043e6761c5718', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.47'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::3dd'}], 'enable_snat': True}, 'gw_port_id': '861b2ae6-e612-419a-a9f7-1846b04360bb', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.47', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::3dd', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:23:21Z', 'updated_at': '2026-06-08T14:23:28Z', 'revision_number': 3, 'tenant_id': '2894d8d8f05c4913a22043e6761c5718'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.762397 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, columns={'external_ids': {'neutron:router_name': 'tempest-ServerBootFromVolumeStableRescueTest-1010071599-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.762558 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "861b2ae6-e612-419a-a9f7-1846b04360bb"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.762672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "861b2ae6-e612-419a-a9f7-1846b04360bb"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.762772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.763330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:59.763330 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.763330 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:59.763330 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:59.763330 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.763330 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:59.763330 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.763330 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:59.763639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router 55f4749b-3181-4ab1-8c76-c41325e4f742. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 55f4749b-3181-4ab1-8c76-c41325e4f742 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:59.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:25:59.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:25:59.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:25:59.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:25:59.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:25:59.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:25:59.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:25:59.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:25:59.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:25:59.764832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:25:59.764832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:25:59.764832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:59.764832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.764832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.764832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:59.764832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:59.764832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.764832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.764832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:59.764832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:59.764832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:59.764832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:59.764832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:59.765136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:59.765136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:59.765136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.765136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:59.765136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:59.765136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.765136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:59.765136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.765136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:59.765136 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 93335fb6-1b48-49ad-8a01-1658f7fe3481 (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:25:59.827555 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, resource={'id': '93335fb6-1b48-49ad-8a01-1658f7fe3481', 'name': 'tempest-TaggedAttachmentsTest-1758790243-router', 'project_id': '143396ea48cb4e8a9aef7b2da4ae9da8', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.221'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::2d6'}], 'enable_snat': True}, 'gw_port_id': 'b4f82672-8864-41ec-8bef-d90ba72b7280', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.221', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::2d6', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:23:26Z', 'updated_at': '2026-06-08T14:23:35Z', 'revision_number': 3, 'tenant_id': '143396ea48cb4e8a9aef7b2da4ae9da8'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.827759 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, columns={'external_ids': {'neutron:router_name': 'tempest-TaggedAttachmentsTest-1758790243-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.827974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "b4f82672-8864-41ec-8bef-d90ba72b7280"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.828154 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "b4f82672-8864-41ec-8bef-d90ba72b7280"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:25:59.828314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.830734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:25:59.830734 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.830734 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:25:59.830734 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:25:59.830734 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.830734 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:25:59.830734 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.830734 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:25:59.830917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router 93335fb6-1b48-49ad-8a01-1658f7fe3481. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.831791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 93335fb6-1b48-49ad-8a01-1658f7fe3481 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.831791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:25:59.831791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:25:59.831791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:25:59.831791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:25:59.831791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:25:59.831791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:25:59.831791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:25:59.831791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:25:59.831791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:25:59.831791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:25:59.832070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:25:59.832070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:25:59.832070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:25:59.832070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.832070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.832070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:25:59.832070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:25:59.832070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:25:59.832070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:25:59.832070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:25:59.832070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:25:59.832070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:25:59.832070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:25:59.832070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:25:59.832373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:25:59.832373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:25:59.832373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:25:59.832373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:25:59.832373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:25:59.832373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:25:59.832373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:25:59.832373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:25:59.832373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:25:59.832373 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 49bacaa1-8ede-4596-ba0a-64ecc3a9a217 (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:26:00.146629 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-49bacaa1-8ede-4596-ba0a-64ecc3a9a217, resource={'id': '49bacaa1-8ede-4596-ba0a-64ecc3a9a217', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:be:b7:0c', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '17e5b0cf-4ee1-4902-9b75-92c18f37aafd', 'device_owner': 'network:router_gateway', 'standard_attr_id': 184, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.129'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::329'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:23:39Z', 'updated_at': '2026-06-08T14:24:06Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.146938 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-49bacaa1-8ede-4596-ba0a-64ecc3a9a217, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1430'}, 'networks': ['172.24.5.129/24', '2001:db8::329/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.147067 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=49bacaa1-8ede-4596-ba0a-64ecc3a9a217, lrouter_port=lrp-49bacaa1-8ede-4596-ba0a-64ecc3a9a217, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.147246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.147674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:00.147674 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.147674 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:00.147674 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:00.147674 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.147674 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:00.147674 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.147674 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:00.148385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 49bacaa1-8ede-4596-ba0a-64ecc3a9a217 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.148385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:00.148385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:26:00.148385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:26:00.148385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:26:00.148385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:26:00.148385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:26:00.148385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:00.148385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.148385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.148385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:00.148613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:00.148613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.148613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.148613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:00.148613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:00.148613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:00.148613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:00.148613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:00.148613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:00.148613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:00.148613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.148613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:00.148613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:00.148613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.148847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:00.148847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.148847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:00.148847 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 98f87ce8-1f5e-4abd-b6de-32f21dbb1d4b (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:26:00.280047 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-98f87ce8-1f5e-4abd-b6de-32f21dbb1d4b, resource={'id': '98f87ce8-1f5e-4abd-b6de-32f21dbb1d4b', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:41:23:8c', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '9c10de0a-4866-44f0-a1a0-4c1a9070072d', 'device_owner': 'network:router_gateway', 'standard_attr_id': 192, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.166'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::106'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:23:41Z', 'updated_at': '2026-06-08T14:23:53Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.280232 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-98f87ce8-1f5e-4abd-b6de-32f21dbb1d4b, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.166/24', '2001:db8::106/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.280712 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=98f87ce8-1f5e-4abd-b6de-32f21dbb1d4b, lrouter_port=lrp-98f87ce8-1f5e-4abd-b6de-32f21dbb1d4b, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.280712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.281152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:00.281152 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.281152 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:00.281152 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:00.281152 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.281152 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:00.281152 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.281152 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:00.282279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 98f87ce8-1f5e-4abd-b6de-32f21dbb1d4b (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.282279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:00.282279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:26:00.282279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:26:00.282279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:26:00.282279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:26:00.282279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:26:00.282279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:00.282279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.282279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.282279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:00.283621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:00.283621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.283621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.283621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:00.283621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:00.283621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:00.283621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:00.283621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:00.283621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:00.283621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:00.283621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.283621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:00.283621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:00.283621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.283879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:00.283879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.283879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:00.283879 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 37b1707e-d3f2-4888-8a3f-ee37f8f220a8 (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:26:00.372005 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-37b1707e-d3f2-4888-8a3f-ee37f8f220a8, resource={'id': '37b1707e-d3f2-4888-8a3f-ee37f8f220a8', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:51:57:d1', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '64129b0c-35f5-448b-874d-cdf90527f8fd', 'device_owner': 'network:router_gateway', 'standard_attr_id': 194, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.36'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::1ae'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:23:42Z', 'updated_at': '2026-06-08T14:24:16Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.372324 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-37b1707e-d3f2-4888-8a3f-ee37f8f220a8, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-64129b0c-35f5-448b-874d-cdf90527f8fd'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.36/24', '2001:db8::1ae/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.372602 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=37b1707e-d3f2-4888-8a3f-ee37f8f220a8, lrouter_port=lrp-37b1707e-d3f2-4888-8a3f-ee37f8f220a8, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.372809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.373449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:00.373449 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.373449 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:00.373449 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:00.373449 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.373449 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:00.373449 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.373449 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:00.374376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 37b1707e-d3f2-4888-8a3f-ee37f8f220a8 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.374376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:00.374376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:26:00.374376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:26:00.374376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:26:00.374376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:26:00.374376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:26:00.374376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:00.374376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.374376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.374376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:00.374866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:00.374866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.374866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.374866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:00.374866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:00.374866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:00.374866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:00.374866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:00.374866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:00.374866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:00.374866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.374866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:00.374866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:00.374866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.375904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:00.375904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.375904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:00.375904 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource b4f82672-8864-41ec-8bef-d90ba72b7280 (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:26:00.453270 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-b4f82672-8864-41ec-8bef-d90ba72b7280, resource={'id': 'b4f82672-8864-41ec-8bef-d90ba72b7280', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:ee:d0:4c', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '93335fb6-1b48-49ad-8a01-1658f7fe3481', 'device_owner': 'network:router_gateway', 'standard_attr_id': 120, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.221'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::2d6'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:23:26Z', 'updated_at': '2026-06-08T14:23:30Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.453592 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-b4f82672-8864-41ec-8bef-d90ba72b7280, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.221/24', '2001:db8::2d6/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.453854 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=b4f82672-8864-41ec-8bef-d90ba72b7280, lrouter_port=lrp-b4f82672-8864-41ec-8bef-d90ba72b7280, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.454158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.454614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:00.454614 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.454614 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:00.454614 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:00.454614 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.454614 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:00.454614 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.454614 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:00.455341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource b4f82672-8864-41ec-8bef-d90ba72b7280 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.455341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:00.455341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:26:00.455341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:26:00.455341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:26:00.455341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:26:00.455341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:26:00.455341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:00.455341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.455341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.455341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:00.455690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:00.455690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.455690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.455690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:00.455690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:00.455690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:00.455690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:00.455690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:00.455690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:00.455690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:00.455690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.455690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:00.455690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:00.455690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.456291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:00.456291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.456291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:00.456291 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 7e30389c-1ed3-4581-91b8-fe46c4ae387f (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:26:00.541368 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-7e30389c-1ed3-4581-91b8-fe46c4ae387f, resource={'id': '7e30389c-1ed3-4581-91b8-fe46c4ae387f', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:35:1e:ad', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '4a3c780d-da5d-48d5-8bb6-d81ad7f30477', 'device_owner': 'network:router_gateway', 'standard_attr_id': 118, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.128'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::24e'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:23:25Z', 'updated_at': '2026-06-08T14:23:30Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.541638 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-7e30389c-1ed3-4581-91b8-fe46c4ae387f, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.128/24', '2001:db8::24e/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.541865 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=7e30389c-1ed3-4581-91b8-fe46c4ae387f, lrouter_port=lrp-7e30389c-1ed3-4581-91b8-fe46c4ae387f, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.542082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.542614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:00.542614 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.542614 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:00.542614 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:00.542614 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.542614 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:00.542614 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.542614 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:00.543320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 7e30389c-1ed3-4581-91b8-fe46c4ae387f (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.543320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:00.543320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:26:00.543320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:26:00.543320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:26:00.543320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:26:00.543320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:26:00.543320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:00.543320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.543320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.543320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:00.543553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:00.543553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.543553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.543553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:00.543553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:00.543553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:00.543553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:00.543553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:00.543553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:00.543553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:00.543553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.543553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:00.543553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:00.543553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.544010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:00.544010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.544010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:00.544010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 861b2ae6-e612-419a-a9f7-1846b04360bb (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:26:00.618637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-861b2ae6-e612-419a-a9f7-1846b04360bb, resource={'id': '861b2ae6-e612-419a-a9f7-1846b04360bb', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:4a:30:88', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '55f4749b-3181-4ab1-8c76-c41325e4f742', 'device_owner': 'network:router_gateway', 'standard_attr_id': 108, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.47'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::3dd'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:23:21Z', 'updated_at': '2026-06-08T14:23:24Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.618818 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-861b2ae6-e612-419a-a9f7-1846b04360bb, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-55f4749b-3181-4ab1-8c76-c41325e4f742'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.47/24', '2001:db8::3dd/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.619037 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=861b2ae6-e612-419a-a9f7-1846b04360bb, lrouter_port=lrp-861b2ae6-e612-419a-a9f7-1846b04360bb, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.619383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.619832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:00.619832 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.619832 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:00.619832 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:00.619832 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.619832 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:00.619832 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.619832 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:00.620545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 861b2ae6-e612-419a-a9f7-1846b04360bb (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.620545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:00.620545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:26:00.620545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:26:00.620545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:26:00.620545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:26:00.620545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:26:00.620545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:00.620545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.620545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.620545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:00.620812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:00.620812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.620812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.620812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:00.620812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:00.620812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:00.620812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:00.620812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:00.620812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:00.620812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:00.620812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.620812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:00.620812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:00.620812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.621229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:00.621229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.621229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:00.621229 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 1605b545-4827-40a9-aff2-5a8c1a322888 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:26:00.621229 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(1605b545-4827-40a9-aff2-5a8c1a322888) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:26:00.626391 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 1a831136-c111-4fdc-b416-b2c554d736d4 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:26:00.626815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(1a831136-c111-4fdc-b416-b2c554d736d4) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:26:00.630641 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 2759f5ee-a2b8-44a0-9fc3-6f341cf26a01 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:26:00.630818 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(2759f5ee-a2b8-44a0-9fc3-6f341cf26a01) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:26:00.637240 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 27fe6e70-95b8-4e17-b03c-66ffe2644210 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:26:00.637427 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(27fe6e70-95b8-4e17-b03c-66ffe2644210) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:26:00.641614 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 30142a7b-2948-414e-974e-65cd3af6d796 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:26:00.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(30142a7b-2948-414e-974e-65cd3af6d796) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:26:00.646038 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 780d3677-f8e0-4a92-8e9a-d92a38e0fd20 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:26:00.646180 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(780d3677-f8e0-4a92-8e9a-d92a38e0fd20) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:26:00.658422 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 7aae6dc0-49eb-43b0-b059-c16021570b3e (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:26:00.658605 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(7aae6dc0-49eb-43b0-b059-c16021570b3e) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:26:00.665148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 981a13f4-9f9f-4403-86bf-cc21b4bb24c4 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:26:00.665305 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(981a13f4-9f9f-4403-86bf-cc21b4bb24c4) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:26:00.675444 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource e7a411ae-9919-4677-8483-f6a3aaf4a6b5 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:26:00.675643 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(e7a411ae-9919-4677-8483-f6a3aaf4a6b5) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:26:00.689095 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource eb7cb55f-2aa3-44ad-bc48-c9cbf4f50160 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:26:00.689499 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(eb7cb55f-2aa3-44ad-bc48-c9cbf4f50160) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:26:00.696542 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_for_inconsistencies finished in 1.838 seconds Jun 08 14:26:00.696784 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:00.696892 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:00.697989 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.698181 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.698278 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.698385 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.698578 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-fcd9b94e-6481-4efa-9d12-3f30e261d2b9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.698578 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.698682 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.699502 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-2025d342-ccdd-4832-8596-444f3a1552c5, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.699502 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-17e5b0cf-4ee1-4902-9b75-92c18f37aafd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.699502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.699659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:00.699659 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.699659 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:00.699659 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:00.699659 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.699659 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:00.699659 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.699659 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:00.700502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.700502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:00.700502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:00.700502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:00.700502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.700502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:00.700502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:00.700502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.700502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.700502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:00.700502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:00.700502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.700872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.700872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:00.700872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:00.700872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:00.700872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:00.700872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:00.700872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:00.700872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:00.700872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.700872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:00.700872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:00.700872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.700872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:00.701162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.701162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:00.701382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.701382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:00.701382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:00.701382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:00.701382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:00.701382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:00.701382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.701382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:00.701382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:00.701382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.701382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:00.701382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:00.701382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.701382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:00.701382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:00.701382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.701702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:00.701702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:00.701702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.701702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:00.701702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:00.701702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:00.701702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.701702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:00.701702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:00.701702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:00.701702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:00.701702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:00.701702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:00.701702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:00.701702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:00.701702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.701702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:00.701702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:00.701702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.702244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:00.702244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.702244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:00.702244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:00.702244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:00.702244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:26:00.703219 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.703219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.703219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:00.703219 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.703219 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:00.703219 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:00.703219 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.703219 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:00.703219 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.703219 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:00.703591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.703591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:00.703591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:00.703591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:00.703591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.703591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:00.703591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:00.703591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:00.703591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.703591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:00.703591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:00.703591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:00.703591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:00.703591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:00.703591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:00.703962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:00.703962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:00.703962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.703962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:00.703962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:00.703962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.703962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:00.703962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.703962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:00.704230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.704230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:00.704230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:00.704230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:00.704230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.704230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:00.704230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:00.704230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:26:00.704230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:00.704230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:00.704230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.704230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.704502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:00.704502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:00.704502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.704502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.704502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:00.704502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:00.704502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:00.704502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:00.704502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:00.704502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:00.704502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:00.704502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.704502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:00.704502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:00.704777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.704777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:00.704777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.704777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:00.705032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.705032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:00.705032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:00.705032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:00.705032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:00.705032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:00.705032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.705032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:00.705032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:00.705032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.705032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:00.705032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:00.705032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.705032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:00.705032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:00.705032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.705412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:00.705412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:00.705412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:26:00.705412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:00.705412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:00.705412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.705412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:00.705412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:00.705412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:00.705412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.705412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:00.705412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:00.705412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:00.705412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:00.705412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:00.705412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:00.705412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:00.705412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:00.705793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.705793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:00.705793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:00.705793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.705793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:00.705793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.705793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:00.705793 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:00.705793 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:00.705793 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:00.706014 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.706014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.706377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:00.706377 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.706377 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:00.706377 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:00.706377 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.706377 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:00.706377 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.706377 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:00.707171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.707171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:00.707171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:00.707171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:00.707171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.707171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:00.707171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:00.707171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:00.707171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.707171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:00.707171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:00.707171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:00.707171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:00.707171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:00.707171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:00.707512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:00.707512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:00.707512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.707512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:00.707512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:00.707512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.707512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:00.707512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.707512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:00.707712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.707712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:00.707712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:00.707712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:00.707712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.707712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:00.707712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:00.707712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:00.707712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:00.707712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.707712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.707972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:00.707972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:00.707972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.707972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.707972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:00.707972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:00.707972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:00.707972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:00.707972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:00.707972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:00.707972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:00.707972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.707972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:00.707972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:00.708237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.708237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:00.708237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.708237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:00.708462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.708462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:00.708462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:00.708462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:00.708462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:00.708462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:00.708462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.708462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:00.708462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:00.708462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.708462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:00.708462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:00.708462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.708462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:00.708462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:00.708462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.708462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:00.708872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:00.708872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:00.708872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:00.708872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.708872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:00.708872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:00.708872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:00.708872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.708872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:00.708872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:00.708872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:00.708872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:00.708872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:00.708872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:00.708872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:00.708872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:00.708872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.708872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:00.708872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:00.709926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.709926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:00.709926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.709926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:00.709926 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:00.709926 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:00.709926 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:00.709926 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.710130 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:00.710130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.710130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:00.710130 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.710130 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:00.710130 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:00.710130 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.710130 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:00.710130 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.710130 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:00.710551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.710551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:00.710551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:00.710551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:00.710551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.710551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:00.710551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:00.710551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.710551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.710551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:00.710551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:00.710551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.710920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:00.710920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:00.710920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:00.710920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:00.710920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:00.710920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:00.710920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:00.710920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:00.710920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.710920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:00.710920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:00.710920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.710920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:00.711841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.711841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:00.711841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.711841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:00.711841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:00.711841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:00.711841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:00.711841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:00.711841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.711841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:00.711841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:00.711841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.711841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:00.711841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:00.711841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.712289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:00.712289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:00.712289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:00.712289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:00.712289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:00.712289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.712289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:00.712289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:00.712289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:00.712289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:00.712289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:00.712289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:00.712289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:00.712289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:00.712289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:00.712289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:00.712289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:00.712289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:00.712289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:00.712819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:00.712819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:00.712819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:00.712819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:00.712819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:00.712819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:03.626565 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:03.628541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:03.628541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:03.628541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:03.667010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:03.667326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:03.669219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:03.669219 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:03.669219 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:03.669219 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:03.669219 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:03.669219 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:03.669219 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:03.669219 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:03.669219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:03.669219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:03.669219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:03.669219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:03.669219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:03.669219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:03.669219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:03.669588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:03.669588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:03.669588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:03.669588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:03.669588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:03.669588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:03.669588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:03.669588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:03.669588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:03.669588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:03.669588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:03.669588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:03.669588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:03.669588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:03.669915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:03.669915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:03.669915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:03.669915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:03.669915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:03.669915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:03.670518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:03.670518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:03.670518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:03.670518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:03.670518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:03.670518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:03.670518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:03.670518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:03.670518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:03.670518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:03.670518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:03.670518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:03.670518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:03.670518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:03.670518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:03.670518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:03.670876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:03.670876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:03.670876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:03.670876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:03.670876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:03.670876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:03.670876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:03.670876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:03.670876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:03.670876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:03.670876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:03.670876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:03.670876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:03.670876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:03.670876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:03.670876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:03.670876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:03.670876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:03.670876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:03.671266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:03.671266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:03.671266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:05.697020 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:05.697737 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:05.697737 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:05.698490 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:05.698710 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:05.698839 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:05.698958 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:05.699091 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-fcd9b94e-6481-4efa-9d12-3f30e261d2b9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:05.699197 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:05.699333 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:05.699446 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-2025d342-ccdd-4832-8596-444f3a1552c5, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:05.699604 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-69474760-504b-4ffa-b48a-6b1a739e3d0f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:05.699714 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:05.699826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.700378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:05.700378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:05.700378 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:05.700378 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:05.700378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:05.700378 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:05.700378 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.700378 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:05.701341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.701341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:05.701341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:05.701341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:05.701341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.701341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:05.701341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:05.701341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.701341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:05.701341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:05.701341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:05.701341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.701579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:05.701579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:05.701579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:05.701579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:05.701579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:05.701579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:05.701579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:05.701579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:05.701579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:05.701579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:05.701579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:05.701579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:05.701579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:05.701852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.701852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:05.702138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.702138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:05.702138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:05.702138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:05.702138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:05.702138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:05.702138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.702138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:05.702138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:05.702138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.702138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:05.702138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:05.702138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.702138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:05.702138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:05.702138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.702745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:05.702745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:05.702745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.702745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:05.702745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:05.702745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:05.702745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.702745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:05.702745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:05.702745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:05.702745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:05.702745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:05.702745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:05.702745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:05.702745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:05.702745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:05.702745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:05.702745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:05.702745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:05.703095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:05.703095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.703095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:05.703095 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:05.703095 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:05.703095 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:05.703095 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:26:05.703244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:05.703323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.703839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:05.703839 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:05.703839 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:05.703839 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:05.703839 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:05.703839 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:05.703839 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.703839 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:05.706412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.706412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:05.706412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:05.706412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:05.706412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.706412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:05.706412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:05.706412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:05.706412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.706412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:05.706412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:05.706412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:05.706412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:05.706412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:05.706412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:05.706834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:05.706834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:05.706834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:05.706834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:05.706834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:05.706834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:05.706834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:05.706834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.706834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:05.707335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:05.708120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.708120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:05.708120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:05.708120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:05.708120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.708120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:05.708120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:05.708120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:26:05.708120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:05.708120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:05.708120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.708120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:05.708344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:05.708344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:05.708344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.708344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:05.708344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:05.708344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:05.708344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:05.708344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:05.708344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:05.708344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:05.708344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:05.708344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:05.708344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:05.708344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:05.708655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:05.708655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:05.708655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.708655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:05.708967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.708967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:05.708967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:05.708967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:05.708967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:05.708967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:05.708967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.708967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:05.708967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:05.708967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.708967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:05.708967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:05.708967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.708967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:05.708967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:05.708967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.709280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:05.709280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:05.709280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:26:05.709280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:05.709280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:05.709280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.709280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:05.709280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:05.709280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:05.709280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.709280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:05.709280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:05.709280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:05.709280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:05.709280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:05.709280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:05.709280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:05.709280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:05.709581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:05.709581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:05.709581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:05.709581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:05.709581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:05.709581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.709581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:05.710091 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:05.710091 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:05.710210 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:05.710272 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:05.710829 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:05.711033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.711460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:05.711460 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:05.711460 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:05.711460 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:05.711460 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:05.711460 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:05.711460 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.711460 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:05.712195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.712195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:05.712195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:05.712195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:05.712195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.712195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:05.712195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:05.712195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:05.712195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.712195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:05.712195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:05.712195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:05.712195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:05.712195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:05.712195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:05.712485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:05.712485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:05.712485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:05.712485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:05.712485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:05.712485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:05.712485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:05.712485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.712485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:05.713090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.713090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:05.713090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:05.713090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:05.713090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.713090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:05.713090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:05.713090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:05.713090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:05.713090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.713090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:05.713319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:05.713319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:05.713319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.713319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:05.713319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:05.713319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:05.713319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:05.713319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:05.713319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:05.713319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:05.713319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:05.713319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:05.713319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:05.713319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:05.713552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:05.713552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:05.713552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.713552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:05.713820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.713820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:05.713820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:05.713820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:05.713820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:05.713820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:05.713820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.713820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:05.713820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:05.713820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.713820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:05.713820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:05.713820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.713820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:05.713820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:05.713820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.713820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:05.714111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:05.714111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:05.714111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:05.714111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.714111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:05.714111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:05.714111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:05.714111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.714111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:05.714111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:05.714111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:05.714111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:05.714111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:05.714111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:05.714111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:05.714111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:05.714111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:05.714111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:05.714111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:05.714710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:05.714710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:05.714710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.714710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:05.714710 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:05.714710 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:05.714710 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:05.714710 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:05.714849 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:05.714887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.715279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:05.715279 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:05.715279 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:05.715279 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:05.715279 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:05.715279 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:05.715279 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.715279 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:05.715811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.715811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:05.715811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:05.715811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:05.715811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.715811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:05.715811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:05.715811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.715811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:05.715811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:05.715811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:05.715811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:05.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:05.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:05.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:05.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:05.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:05.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:05.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:05.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:05.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:05.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:05.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:05.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:05.716269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.716269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:05.716448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.716448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:05.716448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:05.716448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:05.716448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:05.716448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:05.716448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.716448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:05.716448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:05.716448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.716448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:05.716448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:05.716448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.716448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:05.716448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:05.716448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:05.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:05.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:05.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:05.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:05.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:05.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:05.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:05.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:05.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:05.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:05.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:05.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:05.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:05.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:05.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:05.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:05.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:05.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:05.717381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:05.717381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:05.717381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:08.627400 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:08.629455 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:08.629518 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:08.630174 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:08.667643 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:08.667643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:08.667848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:08.667848 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:08.667848 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:08.667848 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:08.667848 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:08.667848 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:08.667848 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:08.667848 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:08.668567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:08.668567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:08.668567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:08.668567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:08.668567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:08.668567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:08.668567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:08.668567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:08.668567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:08.668567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:08.668567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:08.668567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:08.668994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:08.668994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:08.668994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:08.668994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:08.668994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:08.668994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:08.668994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:08.668994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:08.668994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:08.668994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:08.668994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:08.668994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:08.668994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:08.669375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:08.669375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:08.669375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:08.669375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:08.669375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:08.669375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:08.669375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:08.669375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:08.669375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:08.669375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:08.669375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:08.669375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:08.669375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:08.669375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:08.669375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:08.669763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:08.669763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:08.669763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:08.669763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:08.669763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:08.669763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:08.669763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:08.669763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:08.669763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:08.669763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:08.669763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:08.669763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:08.669763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:08.669763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:08.669763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:08.669763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:08.669763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:08.669763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:08.669763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:08.670272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:08.670272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:08.670272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:08.670272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:08.670272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:08.670272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:10.697420 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:10.697907 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:10.698046 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:10.699342 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:10.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:10.699651 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:10.699776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:10.699884 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-fcd9b94e-6481-4efa-9d12-3f30e261d2b9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:10.699984 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a3c780d-da5d-48d5-8bb6-d81ad7f30477, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:10.700091 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-2025d342-ccdd-4832-8596-444f3a1552c5, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:10.700193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-69474760-504b-4ffa-b48a-6b1a739e3d0f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:10.700310 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-93335fb6-1b48-49ad-8a01-1658f7fe3481, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:10.700414 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:10.700514 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1fa4b39-0412-4944-98a2-5dbc0b305e58, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:10.700611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.701117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:10.701117 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:10.701117 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:10.701117 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:10.701117 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:10.701117 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:10.701117 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.701117 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:10.702129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.702129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:10.702129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:10.702129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:10.702129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.702129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:10.702129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:10.702129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.702129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:10.702129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:10.702129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:10.702129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.702566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:10.702566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:10.702566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:10.702566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:10.702566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:10.702566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:10.702566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:10.702566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:10.702566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:10.702566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:10.702566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:10.702566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:10.702566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:10.702971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.702971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:10.703602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.703602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:10.703602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:10.703602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:10.703602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:10.703602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:10.703602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.703602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:10.703602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:10.703602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.703602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:10.703602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:10.703602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.703602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:10.703602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:10.703602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.704135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:10.704135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:10.704135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.704135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:10.704135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:10.704135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:10.704135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.704135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:10.704135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:10.704135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:10.704135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:10.704135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:10.704135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:10.704135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:10.704135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:10.704135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:10.704135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:10.704135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:10.704135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:10.704764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:10.704764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.704764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:10.704764 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:10.704764 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:10.704764 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:10.704764 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:26:10.705843 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:10.706013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.708066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:10.708066 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:10.708066 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:10.708066 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:10.708066 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:10.708066 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:10.708066 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.708066 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:10.708847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.708847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:10.708847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:10.708847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:10.708847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.708847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:10.708847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:10.708847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:10.708847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.708847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:10.708847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:10.708847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:10.708847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:10.708847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:10.708847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:10.709381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:10.709381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:10.709381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:10.709381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:10.709381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:10.709381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:10.709381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:10.709381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.709381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:10.711458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.711458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:10.711458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:10.711458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:10.711458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.711458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:10.711458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:10.711458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:26:10.711458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:10.711458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:10.711458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.711458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:10.711704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:10.711704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:10.711704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.711704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:10.711704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:10.711704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:10.711704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:10.711704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:10.711704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:10.711704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:10.711704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:10.711704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:10.711704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:10.711704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:10.712011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:10.712011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:10.712011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.712011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:10.712011 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:10.712847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.712847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:10.712847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:10.712847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:10.712847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:10.712847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:10.712847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.712847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:10.712847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:10.712847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.712847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:10.712847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:10.712847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.712847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:10.712847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:10.712847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.713117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:10.713117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:10.713117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:26:10.713117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:10.713117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:10.713117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.713117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:10.713117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:10.713117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:10.713117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.713117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:10.713117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:10.713117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:10.713117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:10.713117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:10.713117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:10.713117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:10.713117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:10.713420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:10.713420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:10.713420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:10.713420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:10.713420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:10.713420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.713420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:10.713420 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:10.713420 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:10.713420 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:10.713813 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:10.714006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.714449 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:10.714646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:10.714646 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:10.714646 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:10.714646 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:10.714646 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:10.714646 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:10.714646 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.714646 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:10.715422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.715422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:10.715422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:10.715422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:10.715422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.715422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:10.715422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:10.715422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:10.715422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.715422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:10.715422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:10.715422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:10.715422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:10.715422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:10.715422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:10.715867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:10.715867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:10.715867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:10.715867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:10.715867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:10.715867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:10.715867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:10.715867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.715867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:10.717345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.717345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:10.717345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:10.717345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:10.717345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.717345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:10.717345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:10.717345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:10.717345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:10.717345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.717345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:10.717736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:10.717736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:10.717736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.717736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:10.717736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:10.717736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:10.717736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:10.717736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:10.717736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:10.717736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:10.717736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:10.717736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:10.717736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:10.717736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:10.718225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:10.718225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:10.718225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.718225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:10.719086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.719086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:10.719086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:10.719086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:10.719086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:10.719086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:10.719086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.719086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:10.719086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:10.719086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.719086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:10.719086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:10.719086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.719086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:10.719086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:10.719086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.719086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:10.719406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:10.719406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:10.719406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:10.719406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.719406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:10.719406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:10.719406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:10.719406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.719406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:10.719406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:10.719406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:10.719406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:10.719406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:10.719406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:10.719406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:10.719406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:10.719406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:10.719406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:10.719406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:10.719748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:10.719748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:10.719748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.719748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:10.719748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:10.719748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:10.719748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:10.720677 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:10.720947 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:10.721222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.721806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:10.721806 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:10.721806 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:10.721806 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:10.721806 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:10.721806 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:10.721806 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.721806 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:10.722668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.722668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:10.722668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:10.722668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:10.722668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.722668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:10.722668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:10.722668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.722668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:10.722668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:10.722668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:10.722668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.722939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:10.722939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:10.722939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:10.722939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:10.722939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:10.722939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:10.722939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:10.722939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:10.722939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:10.722939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:10.722939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:10.722939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:10.722939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:10.723171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.723171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:10.723660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.723660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:10.723660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:10.723660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:10.723660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:10.723660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:10.723660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.723660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:10.723660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:10.723660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.723660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:10.723660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:10.723660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.723660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:10.723660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:10.723660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:10.723952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:10.723952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:10.723952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.723952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:10.723952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:10.723952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:10.723952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:10.723952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:10.723952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:10.723952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:10.723952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:10.723952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:10.723952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:10.723952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:10.723952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:10.723952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:10.723952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:10.723952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:10.723952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:10.724337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:10.724337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:10.724337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:13.629681 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:13.630309 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:13.630309 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:13.631033 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:13.694470 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:13.694470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:13.694916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:13.694916 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:13.694916 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:13.694916 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:13.694916 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:13.694916 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:13.694916 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:13.694916 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:13.695844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.065 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:13.695844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:13.695844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:13.695844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:13.695844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:13.695844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:13.695844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:13.695844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:13.695844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:13.695844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:13.695844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:13.695844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:13.696275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:13.696275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:13.696275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:13.696275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:13.696275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:13.696275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:13.696275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:13.696275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:13.696275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:13.696275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:13.696275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:13.696275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:13.696275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:13.696650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:13.696650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:13.696724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:13.696724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:13.696724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:13.696724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:13.696724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:13.696724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:13.696724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:13.696724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:13.696724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:13.696724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:13.696724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:13.696724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:13.696724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:13.696724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:13.696724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:13.696724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:13.697196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:13.697196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:13.697196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:13.697196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:13.697196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:13.697196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:13.697196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:13.697196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:13.697196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:13.697196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:13.697196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:13.697196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:13.697196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:13.697196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:13.697196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:13.697196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:13.697196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:13.697196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:13.697196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:13.697736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:13.697736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:13.697736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:15.697938 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:15.699589 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:15.699708 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:15.700861 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:15.701092 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:15.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:15.701325 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-64129b0c-35f5-448b-874d-cdf90527f8fd, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:15.701428 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-fcd9b94e-6481-4efa-9d12-3f30e261d2b9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:15.701523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2025d342-ccdd-4832-8596-444f3a1552c5, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:15.701620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-69474760-504b-4ffa-b48a-6b1a739e3d0f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:15.701719 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:15.701813 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:15.701908 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1fa4b39-0412-4944-98a2-5dbc0b305e58, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:15.702039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.702575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:15.702575 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:15.702575 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:15.702575 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:15.702575 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:15.702575 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:15.702575 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.702575 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:15.703418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.703418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:15.703418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:15.703418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:15.703418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.703418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:15.703418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:15.703418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.703418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:15.703418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:15.703418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:15.703418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.703648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:15.703648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:15.703648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:15.703648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:15.703648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:15.703648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:15.703648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:15.703648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:15.703648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:15.703648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:15.703648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:15.703648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:15.703648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:15.703877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.703877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:15.704147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.704147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:15.704147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:15.704147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:15.704147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:15.704147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:15.704147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.704147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:15.704147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:15.704147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.704147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:15.704147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:15.704147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.704147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:15.704147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:15.704147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.704657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:15.704657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:15.704657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.704657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:15.704657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:15.704657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:15.704657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.704657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:15.704657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:15.704657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:15.704657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:15.704657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:15.704657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:15.704657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:15.704657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:15.704657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:15.704657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:15.704657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:15.704657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:15.705244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:15.705244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.705244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:15.705244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:15.705244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:15.706028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:15.706159 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:26:15.706470 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:15.706625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.707020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:15.707020 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:15.707020 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:15.707020 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:15.707020 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:15.707020 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:15.707020 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.707020 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:15.707570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.707570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:15.707570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:15.707570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:15.707570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.707570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:15.707570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:15.707570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:15.707570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.707570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:15.707570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:15.707570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:15.707570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:15.707570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:15.707570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:15.708043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:15.708043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:15.708043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:15.708043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:15.708043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:15.708043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:15.708043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:15.708043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.708043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:15.708392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.708392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:15.708392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:15.708392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:15.708392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.708392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:15.708392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:15.708392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:26:15.708392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:15.708392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:15.708392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.708392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:15.708797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:15.708797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:15.708797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.708797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:15.708797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:15.708797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:15.708797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:15.708797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:15.708797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:15.708797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:15.708797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:15.708797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:15.708797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:15.708797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:15.709242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:15.709242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:15.709242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.709242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:15.709242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.709242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:15.709242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:15.709242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:15.709242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:15.709242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:15.709242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.709242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:15.709242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:15.709242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.709723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:15.709723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:15.709723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.709723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:15.709723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:15.709723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.709723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:15.709723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:15.709723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:26:15.709723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:15.709723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:15.709723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.709723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:15.709723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:15.709723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:15.709723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.709723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:15.709723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:15.709723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:15.710331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:15.710331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:15.710331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:15.710331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:15.710331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:15.710331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:15.710331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:15.710331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:15.710331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:15.710331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:15.710331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.710331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:15.713180 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:15.714109 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:15.714186 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:15.714317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:15.714624 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:15.714912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.716300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:15.716300 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:15.716300 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:15.716300 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:15.716300 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:15.716300 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:15.716300 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.716300 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:15.716872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.716872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:15.716872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:15.716872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:15.716872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.716872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:15.716872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:15.716872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:15.716872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.716872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:15.716872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:15.716872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:15.716872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:15.716872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:15.716872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:15.717388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:15.717388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:15.717388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:15.717388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:15.717388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:15.717388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:15.717388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:15.717388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.717388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:15.717683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.717683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:15.717683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:15.717683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:15.717683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.717683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:15.717683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:15.717683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:15.717683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:15.717683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.717683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:15.718031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:15.718031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:15.718031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.718031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:15.718031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:15.718031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:15.718031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:15.718031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:15.718031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:15.718031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:15.718031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:15.718031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:15.718031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:15.718031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:15.718484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:15.718484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:15.718484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.718484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:15.718484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.718484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:15.718484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:15.718484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:15.718484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:15.718484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:15.718484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.718484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:15.718484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:15.718484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.718913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:15.718913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:15.718913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.718913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:15.718913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:15.718913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.718913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:15.718913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:15.718913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:15.718913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:15.718913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.718913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:15.718913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:15.718913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:15.718913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.718913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:15.718913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:15.718913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:15.718913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:15.719524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:15.719524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:15.719524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:15.719524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:15.719524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:15.719524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:15.719524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:15.719524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:15.719524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:15.719524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.719524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:15.719524 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:15.719914 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:15.719914 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:15.719914 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:15.720198 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:15.720398 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:15.720606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.722510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:15.722510 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:15.722510 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:15.722510 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:15.722510 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:15.722510 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:15.722510 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.722510 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:15.723451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.723451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:15.723451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:15.723451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:15.723451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.723451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:15.723451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:15.723451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.723451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:15.723451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:15.723451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:15.723451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.723795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:15.723795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:15.723795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:15.723795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:15.723795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:15.723795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:15.723795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:15.723795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:15.723795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:15.723795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:15.723795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:15.723795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:15.723795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:15.724216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.724216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:15.724303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.724303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:15.724303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:15.724303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:15.724303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:15.724303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:15.724303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.724303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:15.724303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:15.724303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.724303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:15.724303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:15.724303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.724303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:15.724303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:15.724303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:15.724764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:15.724764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:15.724764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.724764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:15.724764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:15.724764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:15.724764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:15.724764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:15.724764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:15.724764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:15.724764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:15.724764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:15.724764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:15.724764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:15.724764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:15.724764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:15.724764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:15.724764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:15.724764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:15.725242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:15.725242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:15.725242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:18.630086 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:18.630640 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:18.630696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:18.631320 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:18.670620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:18.670814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:18.671270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:18.671270 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:18.671270 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:18.671270 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:18.671270 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:18.671270 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:18.671270 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:18.671270 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:18.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:18.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:18.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:18.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:18.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:18.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:18.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:18.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:18.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:18.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:18.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:18.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:18.672325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:18.672325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:18.672325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:18.672325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:18.672325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:18.672325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:18.672325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:18.672325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:18.672325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:18.672325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:18.672325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:18.672325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:18.672325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:18.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:18.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:18.672750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:18.672750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:18.672750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:18.672750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:18.672750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:18.672750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:18.672750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:18.672750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:18.672750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:18.672750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:18.672750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:18.672750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:18.672750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:18.672750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:18.672750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:18.672750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:18.673206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:18.673206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:18.673206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:18.673206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:18.673206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:18.673206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:18.673206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:18.673206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:18.673206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:18.673206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:18.673206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:18.673206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:18.673206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:18.673206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:18.673206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:18.673206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:18.673206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:18.673206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:18.673206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:18.673639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:18.673639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:18.673639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:20.699568 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:20.700044 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:20.700044 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:20.701267 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:20.701267 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:20.701267 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-9c10de0a-4866-44f0-a1a0-4c1a9070072d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:20.701267 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-fcd9b94e-6481-4efa-9d12-3f30e261d2b9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:20.701474 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2025d342-ccdd-4832-8596-444f3a1552c5, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:20.701474 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-69474760-504b-4ffa-b48a-6b1a739e3d0f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:20.701474 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:20.701603 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1fa4b39-0412-4944-98a2-5dbc0b305e58, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:20.701652 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:20.702421 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ba2b93c-4f6a-4134-a3a6-d65bc4e48afa, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:20.702421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.702421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:20.702421 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:20.702421 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:20.702421 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:20.702421 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:20.702421 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:20.702421 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.702421 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:20.702798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.702798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:20.702798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:20.702798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:20.702798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.702798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:20.702798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:20.702798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.702798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:20.702798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:20.702798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:20.702798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.703210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:20.703210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:20.703210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:20.703210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:20.703210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:20.703210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:20.703210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:20.703210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:20.703210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:20.703210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:20.703210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:20.703210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:20.703210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:20.703645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.703645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:20.703645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.703645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:20.703645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:20.703645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:20.703645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:20.703645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:20.703645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.703645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:20.703645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:20.703645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.703645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:20.703645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:20.703645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.704144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:20.704144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:20.704144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.704144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:20.704144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:20.704144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.704144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:20.704144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:20.704144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:20.704144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.704144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:20.704144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:20.704144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:20.704144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:20.704144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:20.704144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:20.704144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:20.704144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:20.704144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:20.704691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:20.704691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:20.704691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:20.704691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:20.704691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.704691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:20.706035 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:20.706035 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:20.706035 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:20.706035 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:26:20.706035 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:20.706035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.706326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:20.706326 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:20.706326 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:20.706326 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:20.706326 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:20.706326 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:20.706326 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.706326 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:20.707892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.707892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:20.707892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:20.707892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:20.707892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.707892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:20.707892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:20.707892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:20.707892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.707892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:20.707892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:20.707892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:20.707892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:20.707892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:20.707892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:20.708433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:20.708433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:20.708433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:20.708433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:20.708433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:20.708433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:20.708433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:20.708433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.708433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:20.708433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.708433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:20.708433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:20.708433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:20.708433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.708904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:20.708904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:20.708904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:26:20.708904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:20.708904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:20.708904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.708904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:20.708904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:20.708904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:20.708904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.708904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:20.708904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:20.708904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:20.709328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:20.709328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:20.709328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:20.709328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:20.709328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:20.709328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:20.709328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:20.709328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:20.709328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:20.709328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:20.709328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.709328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:20.709715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.709715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:20.709715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:20.709715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:20.709715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:20.709715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:20.709715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.709715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:20.709715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:20.709715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.709715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:20.709715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:20.709715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.709715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:20.709715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:20.709715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.710175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:20.710175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:20.710175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:26:20.710175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:20.710175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:20.710175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.710175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:20.710175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:20.710175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:20.710175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.710175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:20.710175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:20.710175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:20.710175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:20.710175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:20.710175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:20.710175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:20.710175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:20.710823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:20.710823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:20.710823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:20.710823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:20.710823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:20.710823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.710823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:20.715331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:20.715331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:20.715331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:20.715331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:20.715331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:20.715331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.715610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:20.715610 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:20.715610 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:20.715610 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:20.715610 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:20.715610 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:20.715610 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.715610 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:20.715610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.715610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:20.715610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:20.715610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:20.715610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.715610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:20.715610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:20.715610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:20.715610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.715610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:20.715921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:20.715921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:20.715921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:20.715921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:20.715921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:20.715921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:20.715921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:20.715921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:20.715921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:20.715921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:20.715921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:20.715921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:20.715921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.715921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:20.716169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.716169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:20.716169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:20.716169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:20.716169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.716169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:20.716169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:20.716169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:20.716169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:20.716169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.716169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:20.716412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:20.716412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:20.716412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.716412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:20.716412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:20.716412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:20.716412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:20.716412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:20.716412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:20.716412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:20.716412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:20.716412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:20.716412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:20.716412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:20.716639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:20.716639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:20.716639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.716639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:20.716721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.716721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:20.716721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:20.716721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:20.716721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:20.716721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:20.716721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.716721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:20.716721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:20.716721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.716721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:20.716721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:20.716721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.716721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:20.716721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:20.716721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.716721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:20.717018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:20.717018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:20.717018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:20.717018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.717018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:20.717018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:20.717018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:20.717018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.717018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:20.717018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:20.717018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:20.717018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:20.717018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:20.717018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:20.717018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:20.717018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:20.717018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:20.717018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:20.717018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:20.717369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:20.717369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:20.717369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.717369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:20.719814 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:20.720083 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:20.720182 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:20.720326 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:20.720571 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:20.720739 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:20.720950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.721315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:20.721315 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:20.721315 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:20.721315 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:20.721315 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:20.721315 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:20.721315 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.721315 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:20.721832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.721832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:20.721832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:20.721832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:20.721832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.721832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:20.721832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:20.721832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.721832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:20.721832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:20.721832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:20.721832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.722092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:20.722092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:20.722092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:20.722092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:20.722092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:20.722092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:20.722092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:20.722092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:20.722092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:20.722092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:20.722092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:20.722092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:20.722092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:20.722322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.722322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:20.722527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.722527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:20.722527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:20.722527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:20.722527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:20.722527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:20.722527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.722527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:20.722527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:20.722527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.722527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:20.722527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:20.722527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.722527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:20.722527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:20.722527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:20.722823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:20.722823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:20.722823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.722823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:20.722823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:20.722823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:20.722823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:20.722823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:20.722823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:20.722823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:20.722823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:20.722823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:20.722823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:20.722823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:20.722823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:20.722823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:20.722823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:20.722823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:20.722823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:20.723168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:20.723168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:20.723168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:23.630525 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:23.631182 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:23.631182 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:23.631783 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:23.674863 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:23.674863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:23.675322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:23.675322 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:23.675322 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:23.675322 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:23.675322 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:23.675322 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:23.675322 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:23.675322 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:23.676004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:23.676004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:23.676004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:23.676004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:23.676004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:23.676004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:23.676004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:23.676004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:23.676004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:23.676004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:23.676004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:23.676004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:23.677236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:23.677236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:23.677236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:23.677236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:23.677236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:23.677236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:23.677236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:23.677236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:23.677236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:23.677236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:23.677236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:23.677236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:23.677236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:23.678501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:23.678501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:23.678501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:23.678501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:23.678501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:23.678501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:23.678501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:23.678501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:23.678501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:23.678501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:23.678501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:23.678501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:23.678501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:23.678501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:23.678501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:23.678749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:23.678749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:23.678749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:23.678749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:23.678749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:23.678749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:23.678749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:23.678749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:23.678749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:23.678749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:23.678749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:23.678749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:23.678749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:23.678749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:23.678749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:23.678749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:23.678749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:23.678749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:23.678749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:23.679051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:23.679051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:23.679051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:23.679051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:23.679051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:23.679051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:25.699902 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:25.700492 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:25.700492 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:25.701685 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:25.701891 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:25.702160 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fcd9b94e-6481-4efa-9d12-3f30e261d2b9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:25.703770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2025d342-ccdd-4832-8596-444f3a1552c5, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:25.703770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-69474760-504b-4ffa-b48a-6b1a739e3d0f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:25.703770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:25.703770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1fa4b39-0412-4944-98a2-5dbc0b305e58, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:25.703770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:25.704099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ba2b93c-4f6a-4134-a3a6-d65bc4e48afa, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:25.704099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.704099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:25.704099 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:25.704099 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:25.704099 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:25.704099 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:25.704099 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:25.704099 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.704099 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:25.704654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.704654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:25.704654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:25.704654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:25.704654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.704654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:25.704654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:25.704654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.704654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:25.704654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:25.704654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:25.704654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.705737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:25.705737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:25.705737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:25.705737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:25.705737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:25.705737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:25.705737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:25.705737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:25.705737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:25.705737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:25.705737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:25.705737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:25.705737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:25.706321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.706321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:25.706321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.706321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:25.706321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:25.706321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:25.706321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:25.706321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:25.706321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.706321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:25.706321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:25.706321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.706321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:25.706321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:25.706321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.706604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:25.706604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:25.706604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.706604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:25.706604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:25.706604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.706604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:25.706604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:25.706604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:25.706604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.706604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:25.706604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:25.706604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:25.706604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:25.706604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:25.706604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:25.706604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:25.706604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:25.706604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:25.707062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:25.707062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:25.707062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:25.707062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:25.707062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.707062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:25.707062 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:25.707062 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:25.707062 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:25.707062 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:26:25.708904 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:25.708904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.709179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:25.709179 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:25.709179 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:25.709179 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:25.709179 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:25.709179 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:25.709179 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.709179 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:25.709718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.709718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:25.709718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:25.709718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:25.709718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.709718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:25.709718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:25.709718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:25.709718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.709718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:25.709718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:25.709718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:25.709718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:25.709718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:25.709718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:25.710702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:25.710702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:25.710702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:25.710702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:25.710702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:25.710702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:25.710702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:25.710702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.710702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:25.710702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.710702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:25.710702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:25.710702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:25.710702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.711756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:25.711756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:25.711756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:26:25.711756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:25.711756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:25.711756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.711756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:25.711756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:25.711756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:25.711756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.711756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:25.711756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:25.711756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:25.712114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:25.712114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:25.712114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:25.712114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:25.712114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:25.712114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:25.712114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:25.712114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:25.712114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:25.712114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:25.712114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.712114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:25.712461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.712461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:25.712461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:25.712461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:25.712461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:25.712461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:25.712461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.712461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:25.712461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:25.712461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.712461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:25.712461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:25.712461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.712461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:25.712461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:25.712461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.712996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:25.712996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:25.712996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:26:25.712996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:25.712996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:25.712996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.712996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:25.712996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:25.712996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:25.712996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.712996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:25.712996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:25.712996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:25.712996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:25.712996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:25.712996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:25.712996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:25.712996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:25.714456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:25.714456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:25.714456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:25.714456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:25.714456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:25.714456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.714456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:25.714456 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:25.715151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:25.715151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:25.715151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:25.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:25.716798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.716939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:25.716939 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:25.716939 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:25.716939 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:25.716939 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:25.716939 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:25.716939 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.716939 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:25.717486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.717486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:25.717486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:25.717486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:25.717486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.717486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:25.717486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:25.717486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:25.717486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.717486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:25.717486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:25.717486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:25.717486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:25.717486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:25.717486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:25.718744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:25.718744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:25.718744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:25.718744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:25.718744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:25.718744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:25.718744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:25.718744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.718744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:25.718744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.718744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:25.718744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:25.718744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:25.718744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.719811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:25.719811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:25.719811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:25.719811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:25.719811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.719811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:25.719811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:25.719811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:25.719811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.719811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:25.719811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:25.719811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:25.719811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:25.720218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:25.720218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:25.720218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:25.720218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:25.720218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:25.720218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:25.720218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:25.720218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:25.720218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:25.720218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.720218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:25.720218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.720538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:25.720538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:25.720538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:25.720538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:25.720538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:25.720538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.720538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:25.720538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:25.720538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.720538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:25.720538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:25.720538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.720538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:25.720538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:25.720538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.720538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:25.720538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:25.720538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:25.720538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.721081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:25.721639 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:25.721639 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:25.721639 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:25.721639 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:25.723335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:25.723335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:25.723770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.724474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:25.724474 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:25.724474 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:25.724474 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:25.724474 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:25.724474 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:25.724474 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.724474 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:25.725124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.725124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:25.725124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:25.725124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:25.725124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.725124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:25.725124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:25.725124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.725124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:25.725124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:25.725124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:25.725124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.725531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:25.725531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:25.725531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:25.725531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:25.725531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:25.725531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:25.725531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:25.725531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:25.725531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:25.725531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:25.725531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:25.725531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:25.725531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:25.725901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.725901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:25.725901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.725901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:25.725901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:25.725901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:25.725901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:25.725901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:25.725901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.725901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:25.725901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:25.725901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.725901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:25.725901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:25.725901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.726407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:25.726407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:25.726407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:25.726407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:25.726407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:25.726407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.726407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:25.726407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:25.726407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:25.726407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:25.726407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:25.726407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:25.726407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:25.726407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:25.726407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:25.726407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:25.726407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:25.726407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:25.726407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:25.726891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:25.726891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:25.726891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:25.726891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:25.726891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:25.726891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:28.631150 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:28.632193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:28.632193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:28.637287 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:28.694526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:28.694747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:28.696615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:28.696615 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:28.696615 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:28.696615 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:28.696615 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:28.696615 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:28.696615 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:28.696615 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:28.697632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.064 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:28.697632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:28.697632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:28.697632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:28.697632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:28.697632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:28.697632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:28.697632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:28.697632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:28.697632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:28.697632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:28.697632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:28.698163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:28.698163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:28.698163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:28.698163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:28.698163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:28.698163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:28.698163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:28.698163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:28.698163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:28.698163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:28.698163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:28.698163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:28.698163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:28.698708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:28.698708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:28.699605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:28.699605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:28.699605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:28.699605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:28.699605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:28.699605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:28.699605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:28.699605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:28.699605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:28.699605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:28.699605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:28.699605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:28.699605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:28.699605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:28.699605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:28.699605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:28.699933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:28.699933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:28.699933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:28.699933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:28.699933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:28.699933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:28.699933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:28.699933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:28.699933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:28.699933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:28.699933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:28.699933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:28.699933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:28.699933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:28.699933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:28.699933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:28.699933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:28.699933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:28.699933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:28.700298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:28.700298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:28.700298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:30.700607 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:30.701422 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:30.701580 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:30.705075 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:30.705245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:30.705397 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fcd9b94e-6481-4efa-9d12-3f30e261d2b9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:30.705513 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2025d342-ccdd-4832-8596-444f3a1552c5, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:30.705679 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-69474760-504b-4ffa-b48a-6b1a739e3d0f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:30.705831 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:30.705983 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1fa4b39-0412-4944-98a2-5dbc0b305e58, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:30.706162 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:30.706338 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ba2b93c-4f6a-4134-a3a6-d65bc4e48afa, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:30.706774 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:30.707030 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:30.707792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.710306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:30.710306 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:30.710306 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:30.710306 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:30.710306 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:30.710306 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:30.710306 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.710306 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:30.710306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.710306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:30.710306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:30.710306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:30.710306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.710306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:30.710306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:30.710684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.710684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:30.710684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:30.710684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:30.710684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.710684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:30.710684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:30.710684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:30.710684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:30.710684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:30.710684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:30.710684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:30.710684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:30.710684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:30.710998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:30.710998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:30.710998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:30.710998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:30.710998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.710998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:30.710998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.710998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:30.710998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:30.710998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:30.710998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:30.710998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:30.710998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.710998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:30.711353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:30.711791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:30.711791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:30.711791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:30.711791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:30.711791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:30.711791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:30.711791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:30.711791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:30.711791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.711791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:30.711791 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:30.712581 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:30.712581 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:26:30.712975 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:30.713234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:30.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:30.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:30.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:30.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:30.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:30.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:30.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:30.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:30.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:30.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:30.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:30.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:30.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.716041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:30.716588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:30.716588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:30.716588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:30.716588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:30.716588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:30.716588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:30.716588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:30.716588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:30.716588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:30.716588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:30.716588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:30.716588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:30.716588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.716588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:30.716588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.717066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:30.717066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:30.717066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:30.717066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.717066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:30.717066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:30.717066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:26:30.717066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:30.717066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:30.717066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.717066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:30.717066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:30.717066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:30.717357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.717357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:30.717357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:30.717357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:30.717357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:30.717357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:30.717357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:30.717357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:30.717357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:30.717357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:30.717357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:30.717357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:30.717357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:30.717357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:30.717917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.717917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:30.717917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.717917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:30.717917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:30.717917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:30.717917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:30.717917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:30.717917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.717917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:30.717917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:30.717917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.717917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:30.717917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:30.717917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.718330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:30.718330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:30.718330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.718330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:30.718330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:30.718330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:26:30.718330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:30.718330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:30.718330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.718330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:30.718330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:30.718330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:30.718330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.718330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:30.718330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:30.718330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:30.718330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:30.718330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:30.718330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:30.718667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:30.718667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:30.718667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:30.718667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:30.718667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:30.718667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:30.718667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:30.718667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.718667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:30.718667 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:30.719383 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:30.719566 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:30.719746 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:30.720908 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:30.721383 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:30.721670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.724642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:30.724642 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:30.724642 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:30.724642 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:30.724642 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:30.724642 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:30.724642 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.724642 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:30.724642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.724642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:30.724642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:30.724642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:30.724642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.724642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:30.724642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:30.724642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:30.724642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.724642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:30.725216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:30.725216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:30.725216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:30.725216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:30.725216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:30.725216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:30.725216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:30.725216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:30.725216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:30.725216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:30.725216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:30.725216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:30.725216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.725216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:30.725216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.725517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:30.725517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:30.725517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:30.725517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.725517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:30.725517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:30.725517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:30.725517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:30.725517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.725517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:30.725517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:30.725517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:30.725517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.725811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:30.725811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:30.725811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:30.725811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:30.725811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:30.725811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:30.725811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:30.725811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:30.725811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:30.725811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:30.725811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:30.725811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:30.725811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:30.726087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.726087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:30.726087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.726087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:30.726087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:30.726087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:30.726087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:30.726087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:30.726087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.726087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:30.726087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:30.726087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.726087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:30.726087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:30.726087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.726371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:30.726371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:30.726371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.726371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:30.726371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:30.726371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:30.726371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:30.726371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.726371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:30.726371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:30.726371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:30.726371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.726371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:30.726371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:30.726371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:30.726371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:30.726371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:30.726371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:30.726371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:30.726699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:30.726699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:30.726699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:30.726699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:30.726699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:30.726699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:30.726699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.726699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:30.726699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:30.726699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:30.726699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:30.726940 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:30.726940 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:30.726940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.727368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:30.727368 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:30.727368 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:30.727368 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:30.727368 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:30.727368 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:30.727368 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.727368 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:30.730578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.730578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:30.730578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:30.730578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:30.730578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.730578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:30.730578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:30.730578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.730578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:30.730578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:30.730578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:30.730578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.730946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:30.730946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:30.730946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:30.730946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:30.730946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:30.730946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:30.730946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:30.730946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:30.730946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:30.730946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:30.730946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:30.730946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:30.730946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:30.731173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.731173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:30.732475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.732475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:30.732475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:30.732475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:30.732475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:30.732475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:30.732475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.732475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:30.732475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:30.732475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.732475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:30.732475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:30.732475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.732475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:30.732475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:30.732475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:30.732851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:30.732851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:30.732851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.732851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:30.732851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:30.732851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:30.732851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:30.732851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:30.732851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:30.732851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:30.732851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:30.732851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:30.732851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:30.732851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:30.732851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:30.732851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:30.732851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:30.732851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:30.732851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:30.733676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:30.733676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:30.733676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:33.632766 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:33.632766 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:33.632766 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:33.633322 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:33.685888 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:33.688214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:33.688502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:33.688502 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:33.688502 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:33.688502 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:33.688502 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:33.688502 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:33.688502 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:33.688502 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:33.689488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:33.689488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:33.689488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:33.689488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:33.689488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:33.689488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:33.689488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:33.689488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:33.689488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:33.689488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:33.689488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:33.689488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:33.689735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:33.689735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:33.689735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:33.689735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:33.689735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:33.689735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:33.689735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:33.689735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:33.689735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:33.689735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:33.689735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:33.689735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:33.689735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:33.689945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:33.689945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:33.690822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:33.690822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:33.690822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:33.690822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:33.690822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:33.690822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:33.690822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:33.690822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:33.690822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:33.690822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:33.690822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:33.690822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:33.690822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:33.690822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:33.690822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:33.690822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:33.691352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:33.691352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:33.691352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:33.691352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:33.691352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:33.691352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:33.691352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:33.691352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:33.691352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:33.691352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:33.691352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:33.691352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:33.691352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:33.691352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:33.691352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:33.691352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:33.691352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:33.691352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:33.691352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:33.691708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:33.691708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:33.691708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:35.701998 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:35.702629 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:35.702629 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:35.707620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:35.707620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:35.707620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fcd9b94e-6481-4efa-9d12-3f30e261d2b9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:35.707620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2025d342-ccdd-4832-8596-444f3a1552c5, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:35.707620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-69474760-504b-4ffa-b48a-6b1a739e3d0f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:35.707801 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:35.707801 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1fa4b39-0412-4944-98a2-5dbc0b305e58, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:35.707801 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:35.707801 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ba2b93c-4f6a-4134-a3a6-d65bc4e48afa, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:35.707801 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:35.707801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.707978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:35.707978 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:35.707978 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:35.707978 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:35.707978 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:35.707978 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:35.707978 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.707978 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:35.710396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.710396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:35.710396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:35.710396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:35.710396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.710396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:35.710396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:35.710396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.710396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:35.710396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:35.710396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:35.710396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.710828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:35.710828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:35.710828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:35.710828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:35.710828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:35.710828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:35.710828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:35.710828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:35.710828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:35.710828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:35.710828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:35.710828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:35.710828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:35.711187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.711187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:35.711187 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:35.712108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.712108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:35.712108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:35.712108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:35.712108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:35.712108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:35.712108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.712108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:35.712108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:35.712108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.712108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:35.712108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:35.712108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.712108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:35.712108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:35.712108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.712633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:35.712633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:35.712633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.712633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:35.712633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:35.712633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:35.712633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.712633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:35.712633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:35.712633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:35.712633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:35.712633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:35.712633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:35.712633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:35.712633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:35.712633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:35.712633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:35.712633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:35.712633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:35.713225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:35.713225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.713225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:35.713516 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:35.713780 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:35.713860 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:26:35.714609 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:35.714609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.715063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:35.715063 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:35.715063 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:35.715063 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:35.715063 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:35.715063 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:35.715063 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.715063 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:35.715657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.715657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:35.715657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:35.715657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:35.715657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.715657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:35.715657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:35.715657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:35.715657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.715657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:35.715657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:35.715657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:35.715657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:35.715657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:35.715657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:35.715952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:35.715952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:35.715952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:35.715952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:35.715952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:35.715952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:35.715952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:35.715952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.715952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:35.716232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.716232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:35.716232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:35.716232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:35.716232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.716232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:35.716232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:35.716232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:26:35.716232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:35.716232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:35.716232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.716232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:35.716492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:35.716492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:35.716492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.716492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:35.716492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:35.716492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:35.716492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:35.716492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:35.716492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:35.716492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:35.716492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:35.716492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:35.716492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:35.716492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:35.716726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:35.716726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:35.716726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.716726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:35.716938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.716938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:35.716938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:35.716938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:35.716938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:35.716938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:35.716938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.716938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:35.716938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:35.716938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.716938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:35.716938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:35.716938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.716938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:35.716938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:35.716938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.717234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:35.717234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:35.717234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:26:35.717234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:35.717234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:35.717234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.717234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:35.717234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:35.717234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:35.717234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.717234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:35.717234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:35.717234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:35.717234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:35.717234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:35.717234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:35.717234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:35.717234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:35.717643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:35.717643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:35.717643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:35.717643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:35.717643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:35.717643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.717643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:35.719315 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:35.719880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:35.719966 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:35.720079 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:35.720596 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:35.720596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.721231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:35.721231 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:35.721231 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:35.721231 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:35.721231 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:35.721231 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:35.721231 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.721231 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:35.721794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.721794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:35.721794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:35.721794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:35.721794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.721794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:35.721794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:35.721794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:35.721794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.721794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:35.721794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:35.721794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:35.721794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:35.721794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:35.721794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:35.722083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:35.722083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:35.722083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:35.722083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:35.722083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:35.722083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:35.722083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:35.722083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.722083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:35.722534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.722534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:35.722534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:35.722534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:35.722534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.722534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:35.722534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:35.722534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:35.722534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:35.722534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.722534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:35.722740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:35.722740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:35.722740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.722740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:35.722740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:35.722740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:35.722740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:35.722740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:35.722740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:35.722740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:35.722740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:35.722740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:35.722740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:35.722740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:35.723023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:35.723023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:35.723023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.723023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:35.723323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.723323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:35.723323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:35.723323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:35.723323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:35.723323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:35.723323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.723323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:35.723323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:35.723323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.723323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:35.723323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:35.723323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.723323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:35.723323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:35.723323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.723323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:35.723647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:35.723647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:35.723647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:35.723647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.723647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:35.723647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:35.723647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:35.723647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.723647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:35.723647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:35.723647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:35.723647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:35.723647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:35.723647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:35.723647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:35.723647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:35.723647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:35.723647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:35.723647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:35.723975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:35.723975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:35.723975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.723975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:35.724897 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:35.725118 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:35.725198 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:35.725397 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:35.726290 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:35.726290 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:35.726290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.726378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:35.726378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:35.726378 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:35.726378 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:35.726378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:35.726378 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:35.726378 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.726378 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:35.727002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.727002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:35.727002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:35.727002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:35.727002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.727002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:35.727002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:35.727002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.727002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:35.727002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:35.727002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:35.727002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.727321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:35.727321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:35.727321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:35.727321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:35.727321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:35.727321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:35.727321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:35.727321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:35.727321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:35.727321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:35.727321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:35.727321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:35.727321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:35.727543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.727543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:35.727703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.727703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:35.727703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:35.727703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:35.727703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:35.727703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:35.727703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.727703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:35.727703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:35.727703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.727703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:35.727703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:35.727703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.727703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:35.727703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:35.727703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:35.728004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:35.728004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:35.728004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.728004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:35.728004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:35.728004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:35.728004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:35.728004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:35.728004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:35.728004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:35.728004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:35.728004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:35.728004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:35.728004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:35.728004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:35.728004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:35.728004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:35.728004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:35.728004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:35.728425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:35.728425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:35.728425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:38.632563 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:38.634780 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:38.634780 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:38.636212 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:38.696619 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:38.696805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:38.697184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:38.697184 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:38.697184 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:38.697184 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:38.697184 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:38.697184 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:38.697184 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:38.697184 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:38.697840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.062 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:38.697840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:38.697840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:38.697840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:38.697840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:38.697840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:38.697840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:38.697840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:38.697840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:38.697840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:38.697840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:38.697840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:38.698369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:38.698369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:38.698369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:38.698369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:38.698369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:38.698369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:38.698369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:38.698369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:38.698369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:38.698369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:38.698369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:38.698369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:38.698369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:38.699067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:38.699067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:38.699067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:38.699067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:38.699067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:38.699067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:38.699067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:38.699067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:38.699067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:38.699067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:38.699067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:38.699067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:38.699067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:38.699067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:38.699067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:38.699752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:38.699752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:38.699752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:38.699752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:38.699752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:38.699752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:38.699752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:38.699752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:38.699752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:38.699752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:38.699752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:38.699752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:38.699752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:38.699752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:38.699752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:38.699752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:38.699752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:38.699752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:38.699752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:38.700640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:38.700640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:38.700640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:38.700640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:38.700640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:38.700640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:40.702195 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:40.702787 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:40.702787 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:40.703676 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:40.703879 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:40.703990 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fcd9b94e-6481-4efa-9d12-3f30e261d2b9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:40.704129 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2025d342-ccdd-4832-8596-444f3a1552c5, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:40.704286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-69474760-504b-4ffa-b48a-6b1a739e3d0f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:40.704376 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:40.704495 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1fa4b39-0412-4944-98a2-5dbc0b305e58, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:40.704599 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ba2b93c-4f6a-4134-a3a6-d65bc4e48afa, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:40.704696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:40.704808 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:40.704933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.706427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:40.706427 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:40.706427 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:40.706427 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:40.706427 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:40.706427 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:40.706427 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.706427 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:40.707117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.707117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:40.707117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:40.707117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:40.707117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.707117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:40.707117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:40.707117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.707117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:40.707117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:40.707117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:40.707117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.707507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:40.707507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:40.707507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:40.707507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:40.707507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:40.707507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:40.707507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:40.707507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:40.707507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:40.707507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:40.707507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:40.707507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:40.707507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:40.707766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.707766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:40.707816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.707816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:40.707816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:40.707816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:40.707816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:40.707816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:40.707816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.707816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:40.707816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:40.707816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.707816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:40.707816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:40.707816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.707816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:40.707816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:40.707816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.708117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:40.708117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:40.708117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.708117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:40.708117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:40.708117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:40.708117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.708117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:40.708117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:40.708117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:40.708117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:40.708117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:40.708117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:40.708117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:40.708117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:40.708117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:40.708117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:40.708117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:40.708117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:40.708459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:40.708459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.708459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:40.712456 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:40.714775 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:40.715029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:40.715167 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:26:40.719399 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:40.719578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.720100 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:40.720342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:40.720342 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:40.720342 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:40.720342 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:40.720342 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:40.720342 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:40.720342 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.720342 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:40.721118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.721118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:40.721118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:40.721118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:40.721118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.721118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:40.721118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:40.721118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:40.721118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.721118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:40.721118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:40.721118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:40.721118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:40.721118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:40.721118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:40.721879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:40.721879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:40.721879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:40.721879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:40.721879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:40.721879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:40.721879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:40.721879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.721879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:40.721879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.721879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:40.721879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:40.721879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:40.721879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.722326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:40.722326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:40.722326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:26:40.722326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:40.722326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:40.722326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.722326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:40.722326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:40.722326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:40.722326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.722326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:40.722326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:40.722326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:40.722590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:40.722590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:40.722590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:40.722590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:40.722590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:40.722590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:40.722590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:40.722590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:40.722590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:40.722590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:40.722590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.722590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:40.722800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.722800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:40.722800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:40.722800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:40.722800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:40.722800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:40.722800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.722800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:40.722800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:40.722800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.722800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:40.722800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:40.722800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.722800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:40.722800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:40.722800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.723143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:40.723143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:40.723143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:26:40.723143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:40.723143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:40.723143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.723143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:40.723143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:40.723143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:40.723143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.723143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:40.723143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:40.723143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:40.723143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:40.723143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:40.723143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:40.723143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:40.723143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:40.723480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:40.723480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:40.723480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:40.723480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:40.723480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:40.723480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.723480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:40.723480 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:40.723480 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:40.723480 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:40.723664 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:40.723664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.724149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:40.724149 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:40.724149 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:40.724149 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:40.724149 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:40.724149 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:40.724149 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.724149 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:40.725070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.725070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:40.725070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:40.725070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:40.725070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.725070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:40.725070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:40.725070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:40.725070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.725070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:40.725070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:40.725070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:40.725070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:40.725070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:40.725070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:40.725629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:40.725629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:40.725629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:40.725629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:40.725629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:40.725629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:40.725629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:40.725629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.725629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:40.725629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.725629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:40.725629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:40.725629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:40.725629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.725962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:40.725962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:40.725962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:40.725962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:40.725962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.725962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:40.725962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:40.725962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:40.725962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.725962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:40.725962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:40.725962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:40.725962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:40.726184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:40.726184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:40.726184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:40.726184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:40.726184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:40.726184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:40.726184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:40.726184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:40.726184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:40.726184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.726184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:40.726184 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:40.727023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.727023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:40.727023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:40.727023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:40.727023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:40.727023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:40.727023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.727023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:40.727023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:40.727023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.727023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:40.727023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:40.727023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.727023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:40.727023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:40.727023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.727023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:40.727549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:40.727549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:40.727549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:40.727549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.727549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:40.727549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:40.727549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:40.727549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.727549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:40.727549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:40.727549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:40.727549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:40.727549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:40.727549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:40.727549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:40.727549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:40.727549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:40.727549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:40.727549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:40.727835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:40.727835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:40.727835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.727835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:40.727835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:40.727835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:40.727835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:40.727835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:40.727987 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:40.727987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.727987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:40.727987 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:40.727987 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:40.727987 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:40.727987 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:40.727987 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:40.727987 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.727987 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:40.728551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.728551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:40.728551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:40.728551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:40.728551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.728551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:40.728551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:40.728551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.728551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:40.728551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:40.728551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:40.728551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.729001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:40.729001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:40.729001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:40.729001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:40.729001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:40.729001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:40.729001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:40.729001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:40.729001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:40.729001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:40.729001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:40.729001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:40.729001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:40.729628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.729628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:40.729628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.729628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:40.729628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:40.729628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:40.729628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:40.729628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:40.729628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.729628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:40.729628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:40.729628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.729628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:40.729628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:40.729628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.730272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:40.730272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:40.730272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:40.730272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:40.730272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:40.730272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.730272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:40.730272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:40.730272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:40.730272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:40.730272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:40.730272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:40.730272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:40.730272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:40.730272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:40.730272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:40.730272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:40.730272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:40.730272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:40.730957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:40.730957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:40.730957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:40.730957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:40.730957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:40.730957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:43.634662 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:43.636506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:43.636506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:43.636506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:43.697009 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:43.697009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:43.698466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:43.698466 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:43.698466 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:43.698466 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:43.698466 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:43.698466 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:43.698466 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:43.698466 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:43.699775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.063 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:43.699775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:43.699775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:43.699775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:43.699775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:43.699775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:43.699775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:43.699775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:43.699775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:43.699775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:43.699775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:43.699775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:43.700221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:43.700221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:43.700221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:43.700221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:43.700221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:43.700221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:43.700221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:43.700221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:43.700221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:43.700221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:43.700221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:43.700221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:43.700221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:43.700985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:43.700985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:43.700985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:43.700985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:43.700985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:43.700985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:43.700985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:43.700985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:43.700985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:43.700985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:43.700985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:43.700985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:43.700985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:43.700985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:43.700985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:43.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:43.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:43.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:43.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:43.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:43.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:43.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:43.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:43.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:43.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:43.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:43.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:43.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:43.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:43.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:43.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:43.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:43.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:43.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:43.701936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:43.701936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:43.701936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:43.701936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:43.701936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:43.701936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:45.702836 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:45.703331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:45.703331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:45.704043 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:45.704227 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:45.704356 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2025d342-ccdd-4832-8596-444f3a1552c5, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:45.704457 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-69474760-504b-4ffa-b48a-6b1a739e3d0f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:45.704561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1fa4b39-0412-4944-98a2-5dbc0b305e58, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:45.704662 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ba2b93c-4f6a-4134-a3a6-d65bc4e48afa, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:45.704824 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:45.704867 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:45.704958 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:45.705074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.705597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:45.705597 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:45.705597 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:45.705597 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:45.705597 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:45.705597 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:45.705597 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.705597 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:45.706345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.706345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:45.706345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:45.706345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:45.706345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.706345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:45.706345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:45.706345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.706345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:45.706345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:45.706345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:45.706345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.707109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:45.707109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:45.707109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:45.707109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:45.707109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:45.707109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:45.707109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:45.707109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:45.707109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:45.707109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:45.707109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:45.707109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:45.707109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:45.707461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.707461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:45.707461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.707461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:45.707461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:45.707461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:45.707461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:45.707461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:45.707461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.707461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:45.707461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:45.707461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.707461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:45.707461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:45.707461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.707860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:45.707860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:45.707860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.707860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:45.707860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:45.707860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.707860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:45.707860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:45.707860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:45.707860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.707860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:45.707860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:45.707860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:45.707860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:45.707860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:45.707860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:45.707860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:45.707860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:45.707860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:45.708449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:45.708449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:45.708449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:45.708449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:45.708449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.708449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:45.714859 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:45.715401 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:45.715516 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:45.715657 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:26:45.716413 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:45.716594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.718662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:45.718662 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:45.718662 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:45.718662 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:45.718662 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:45.718662 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:45.718662 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.718662 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:45.720094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.720094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:45.720094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:45.720094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:45.720094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.720094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:45.720094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:45.720094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:45.720094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.720094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:45.720094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:45.720094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:45.720094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:45.720094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:45.720094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:45.721008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:45.721008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:45.721008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:45.721008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:45.721008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:45.721008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:45.721008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:45.721008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.721008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:45.721984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.721984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:45.721984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:45.721984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:45.721984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.721984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:45.721984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:45.721984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:26:45.721984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:45.721984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:45.721984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.721984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:45.722591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:45.722591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:45.722591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.722591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:45.722591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:45.722591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:45.722591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:45.722591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:45.722591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:45.722591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:45.722591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:45.722591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:45.722591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:45.722591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:45.723135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:45.723135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:45.723135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.723135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:45.724517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.724517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:45.724517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:45.724517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:45.724517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:45.724517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:45.724517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.724517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:45.724517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:45.724517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.724517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:45.724517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:45.724517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.724517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:45.724517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:45.724517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.725292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:45.725292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:45.725292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:26:45.725292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:45.725292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:45.725292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.725292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:45.725292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:45.725292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:45.725292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.725292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:45.725292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:45.725292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:45.725292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:45.725292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:45.725292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:45.725292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:45.725292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:45.725947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:45.725947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:45.725947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:45.725947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:45.725947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:45.725947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.725947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:45.725947 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:45.725947 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:45.725947 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:45.726278 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:45.726435 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:45.729209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.729209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:45.729209 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:45.729209 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:45.729209 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:45.729209 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:45.729209 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:45.729209 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.729209 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:45.730085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.730085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:45.730085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:45.730085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:45.730085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.730085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:45.730085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:45.730085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:45.730085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.730085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:45.730085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:45.730085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:45.730085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:45.730085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:45.730085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:45.730982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:45.730982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:45.730982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:45.730982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:45.730982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:45.730982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:45.730982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:45.730982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.730982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:45.736262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.736262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:45.736262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:45.736262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:45.736262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.736262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:45.736262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:45.736262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:45.736262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:45.736262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.736262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:45.737282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:45.737282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:45.737282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.737282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:45.737282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:45.737282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:45.737282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:45.737282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:45.737282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:45.737282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:45.737282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:45.737282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:45.737282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:45.737282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:45.738420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:45.738420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:45.738420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.738420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:45.738420 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:45.740472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.740472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:45.740472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:45.740472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:45.740472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:45.740472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:45.740472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.740472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:45.740472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:45.740472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.740472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:45.740472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:45.740472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.740472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:45.740472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:45.740472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.740472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:45.741728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:45.741728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:45.741728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:45.741728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.741728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:45.741728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:45.741728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:45.741728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.741728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:45.741728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:45.741728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:45.741728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:45.741728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:45.741728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:45.741728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:45.741728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:45.741728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:45.741728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:45.741728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:45.742238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:45.742238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:45.742238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.742238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:45.742238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:45.742238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:45.742238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:45.742238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:45.742497 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:45.742497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.742497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:45.742497 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:45.742497 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:45.742497 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:45.742497 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:45.742497 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:45.742497 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.742497 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:45.742497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.742497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:45.742497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:45.742497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:45.742868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.742868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:45.742868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:45.742868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.742868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:45.742868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:45.742868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:45.742868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.742868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:45.742868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:45.742868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:45.742868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:45.742868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:45.742868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:45.744821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:45.744821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:45.744821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:45.744821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:45.744821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:45.744821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:45.744821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:45.744821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.744821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:45.744821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.744821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:45.744821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:45.745587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:45.746950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:45.746950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:45.746950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:45.746950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:45.746950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:45.746950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:45.746950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:45.746950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:45.746950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:45.746950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:45.746950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:45.746950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:45.746950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:45.746950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:45.746950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:48.636491 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:48.636973 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:48.636973 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:48.638692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:48.714529 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:48.714764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:48.715314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:48.715314 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:48.715314 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:48.715314 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:48.715314 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:48.715314 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:48.715314 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:48.715314 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:48.716285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.078 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:48.716285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:48.716285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:48.716285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:48.716285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:48.716285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:48.716285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:48.716285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:48.716285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:48.716285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:48.716285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:48.716285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:48.716581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:48.716581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:48.716581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:48.716581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:48.716581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:48.716581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:48.716581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:48.716581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:48.716581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:48.716581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:48.716581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:48.716581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:48.716581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:48.716828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:48.716828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:48.719279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:48.719279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:48.719279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:48.719279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:48.719279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:48.719279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:48.719279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:48.719279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:48.719279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:48.719279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:48.719279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:48.719279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:48.719279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:48.719279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:48.719279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:48.719279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:48.719799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:48.719799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:48.719799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:48.719799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:48.719799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:48.719799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:48.719799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:48.719799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:48.719799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:48.719799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:48.719799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:48.719799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:48.719799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:48.719799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:48.719799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:48.719799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:48.719799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:48.719799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:48.719799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:48.720229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:48.720229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:48.720229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:50.703347 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:50.703898 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:50.703898 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:50.705446 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:50.705554 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:50.705703 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2025d342-ccdd-4832-8596-444f3a1552c5, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:50.705821 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-69474760-504b-4ffa-b48a-6b1a739e3d0f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:50.705923 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1fa4b39-0412-4944-98a2-5dbc0b305e58, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:50.706044 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ba2b93c-4f6a-4134-a3a6-d65bc4e48afa, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:50.706152 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:50.706266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:50.706361 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:50.706459 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:50.706554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.707802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:50.707802 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:50.707802 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:50.707802 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:50.707802 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:50.707802 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:50.707802 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.707802 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:50.708223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.708223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:50.708223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:50.708223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:50.708223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.708223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:50.708223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:50.708223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.708223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:50.708223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:50.708223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:50.708223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:50.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:50.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:50.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:50.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:50.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:50.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:50.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:50.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:50.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:50.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:50.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:50.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:50.708906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.708906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:50.708906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.708906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:50.708906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:50.708906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:50.708906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:50.708906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:50.708906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.708906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:50.708906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:50.708906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.708906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:50.708906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:50.708906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.709353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:50.709353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:50.709353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.709353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:50.709353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:50.709353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.709353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:50.709353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:50.709353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:50.709353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.709353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:50.709353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:50.709353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:50.709353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:50.709353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:50.709353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:50.709353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:50.709353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:50.709353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:50.709712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:50.709712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:50.709712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:50.709712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:50.709712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.709712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:50.715447 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:50.715792 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:50.716034 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:50.716034 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:26:50.716275 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:50.716431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.717327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:50.717327 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:50.717327 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:50.717327 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:50.717327 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:50.717327 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:50.717327 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.717327 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:50.717327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.717327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:50.717327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:50.717327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:50.717327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.717327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:50.717327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:50.717327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:50.717327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.717327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:50.717709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:50.717709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:50.717709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:50.717709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:50.717709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:50.717709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:50.717709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:50.717709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:50.717709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:50.717709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:50.717709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:50.717709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:50.717709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.717709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:50.718273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.718273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:50.718273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:50.718273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:50.718273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.718273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:50.718273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:50.718273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:26:50.718273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:50.718273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:50.718273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.718273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:50.718541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:50.718541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:50.718541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.718541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:50.718541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:50.718541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:50.718541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:50.718541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:50.718541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:50.718541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:50.718541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:50.718541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:50.718541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:50.718541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:50.718787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:50.718787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:50.718787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.718787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:50.718919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.718919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:50.718919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:50.718919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:50.718919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:50.718919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:50.718919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.718919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:50.718919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:50.718919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.718919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:50.718919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:50.718919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.718919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:50.718919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:50.718919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.719209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:50.719209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:50.719209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:26:50.719209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:50.719209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:50.719209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.719209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:50.719209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:50.719209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:50.719209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.719209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:50.719209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:50.719209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:50.719209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:50.719209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:50.719209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:50.719209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:50.719209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:50.719552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:50.719552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:50.719552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:50.719552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:50.719552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:50.719552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.719552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:50.726284 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:50.726519 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:50.726632 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:50.726708 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:50.728043 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:50.728448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.729387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:50.729387 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:50.729387 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:50.729387 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:50.729387 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:50.729387 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:50.729387 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.729387 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:50.730009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.730009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:50.730009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:50.730009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:50.730009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.730009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:50.730009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:50.730009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:50.730009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.730009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:50.730009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:50.730009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:50.730009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:50.730009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:50.730009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:50.730497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:50.730497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:50.730497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:50.730497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:50.730497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:50.730497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:50.730497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:50.730497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.730497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:50.730665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.730665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:50.730665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:50.730665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:50.730665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.730665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:50.730665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:50.730665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:50.730665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:50.730665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.730665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:50.730871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:50.730871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:50.730871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.730871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:50.730871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:50.730871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:50.730871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:50.730871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:50.730871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:50.730871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:50.730871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:50.730871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:50.730871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:50.730871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:50.731153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:50.731153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:50.731153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.731153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:50.731292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.731292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:50.731292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:50.731292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:50.731292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:50.731292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:50.731292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.731292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:50.731292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:50.731292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.731292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:50.731292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:50.731292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.731292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:50.731292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:50.731292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.731292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:50.731668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:50.731668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:50.731668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:50.731668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.731668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:50.731668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:50.731668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:50.731668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.731668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:50.731668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:50.731668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:50.731668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:50.731668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:50.731668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:50.731668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:50.731668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:50.731668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:50.731668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:50.731668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:50.732161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:50.732161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:50.732161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.732161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:50.740340 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:50.740671 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:50.740784 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:50.740894 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:50.742193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:50.742193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:50.742193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.742907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:50.742907 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:50.742907 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:50.742907 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:50.742907 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:50.742907 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:50.742907 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.742907 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:50.743094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.743094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:50.743094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:50.743094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:50.743094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.743094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:50.743094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:50.743094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.743094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:50.743094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:50.743094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:50.743094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.743359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:50.743359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:50.743359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:50.743359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:50.743359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:50.743359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:50.743359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:50.743359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:50.743359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:50.743359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:50.743359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:50.743359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:50.743359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:50.743596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.743596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:50.744396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.744396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:50.744396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:50.744396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:50.744396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:50.744396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:50.744396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.744396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:50.744396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:50.744396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.744396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:50.744396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:50.744396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.744396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:50.744396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:50.744396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:50.744691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:50.744691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:50.744691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.744691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:50.744691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:50.744691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:50.744691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:50.744691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:50.744691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:50.744691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:50.744691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:50.744691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:50.744691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:50.744691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:50.744691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:50.744691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:50.744691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:50.744691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:50.744691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:50.745035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:50.745035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:50.745035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:53.638011 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:53.640769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:53.640769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:53.640769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:53.643833 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=1114,neutron:UPDATE=1,neutron:DELETE=10 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:26:53.750929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:53.751045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:53.751526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:53.751526 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:53.751526 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:53.751526 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:53.751526 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:53.751526 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:53.751526 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:53.751526 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:53.752845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.113 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:53.752845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:53.752845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:53.752845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:53.752845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:53.752845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:53.752845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:53.752845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:53.752845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:53.752845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:53.752845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:53.752845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:53.753228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:53.753228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:53.753228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:53.753228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:53.753228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:53.753228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:53.753228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:53.753228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:53.753228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:53.753228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:53.753228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:53.753228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:53.753228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:53.753540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:53.753540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:53.757376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:53.757376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:53.757376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:53.757376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:53.757376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:53.757376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:53.757376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:53.757376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:53.757376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:53.757376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:53.757376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:53.757376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:53.757376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:53.757376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:53.757376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:53.757376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:53.757957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:53.757957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:53.757957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:53.757957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:53.757957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:53.757957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:53.757957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:53.757957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:53.757957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:53.757957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:53.757957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:53.757957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:53.757957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:53.757957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:53.757957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:53.757957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:53.757957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:53.757957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:53.757957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:53.758386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:53.758386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:53.758386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:55.703700 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:55.704274 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:55.704274 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:55.707273 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:55.707273 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:55.707273 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1fa4b39-0412-4944-98a2-5dbc0b305e58, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:55.707273 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ba2b93c-4f6a-4134-a3a6-d65bc4e48afa, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:55.707273 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:55.707517 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:55.707517 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:55.707517 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:55.707517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.707517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:55.707517 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:55.707517 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:55.707517 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:55.707517 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:55.707517 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:55.707517 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.707517 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:55.707845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.707845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:55.707845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:55.707845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:55.707845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.707845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:55.707845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:55.707845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.707845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:55.707845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:55.707845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:55.707845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.708091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:55.708091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:55.708091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:55.708091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:55.708091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:55.708091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:55.708091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:55.708091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:55.708091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:55.708091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:55.708091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:55.708091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:55.708091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:55.708371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.708371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:55.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:55.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:55.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:55.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:55.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:55.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:55.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:55.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:55.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:55.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:55.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:55.708647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.709000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:26:55.709000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:55.709000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.709000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:55.709000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:55.709000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:55.709000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.709000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:55.709000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:55.709000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:55.709000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:55.709000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:55.709000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:55.709000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:55.709000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:55.709000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:55.709000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:55.709000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:55.709000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:55.709599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:55.709599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.709599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:55.715886 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:55.718097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:55.718097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:55.718097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:26:55.718097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:55.718097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.718782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:55.718782 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:55.718782 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:55.718782 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:55.718782 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:55.718782 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:55.718782 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.718782 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:55.719345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.719345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:55.719345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:55.719345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:55.719345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.719345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:55.719345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:55.719345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:55.719345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.719345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:55.719345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:55.719345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:55.719345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:55.719345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:55.719345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:55.719826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:55.719826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:55.719826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:55.719826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:55.719826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:55.719826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:55.719826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:55.719826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.719826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:55.720111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.720111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:55.720111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:55.720111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:55.720111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.720111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:55.720111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:55.720111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:26:55.720111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:55.720111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:55.720111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.720111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:55.720606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:55.720606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:55.720606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.720606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:55.720606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:55.720606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:55.720606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:55.720606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:55.720606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:55.720606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:55.720606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:55.720606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:55.720606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:55.720606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:55.720969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:55.720969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:55.720969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.720969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:55.720969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.720969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:55.720969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:55.720969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:55.720969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:55.720969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:55.720969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.720969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:55.720969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:55.720969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.721491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:55.721491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:55.721491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.721491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:55.721491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:55.721491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.721491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:26:55.721491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:26:55.721491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:26:55.721491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:55.721491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:55.721491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.721491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:55.721491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:55.721491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:55.721491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.721491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:55.721491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:55.721491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:55.722039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:55.722039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:55.722039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:55.722039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:55.722039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:55.722039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:55.722039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:55.722039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:55.722039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:55.722039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:55.722039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.722039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:55.726631 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:55.728680 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:55.728779 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:55.728921 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:55.731165 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:55.731393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.731819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:55.731819 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:55.731819 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:55.731819 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:55.731819 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:55.731819 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:55.731819 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.731819 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:55.732375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.732375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:26:55.732375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:55.732375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:55.732375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.732375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:55.732375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:55.732375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:26:55.732375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.732375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:26:55.732375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:55.732375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:26:55.732375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:55.732375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:26:55.732375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:26:55.732825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:55.732825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:26:55.732825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:55.732825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:26:55.732825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:26:55.732825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:55.732825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:26:55.732825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.732825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:26:55.733139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.733139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:55.733139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:55.733139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:55.733139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.733139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:55.733139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:55.733139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:55.733139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:55.733139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.733139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:55.733529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:55.733529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:55.733529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.733529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:55.733529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:55.733529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:55.733529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:55.733529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:55.733529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:55.733529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:55.733529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:55.733529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:55.733529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:55.733529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:55.734034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:55.734034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:55.734034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.734034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:55.734034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.734034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:55.734034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:55.734034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:55.734034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:55.734034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:55.734034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.734034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:55.734034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:55.734034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.734479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:55.734479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:55.734479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.734479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:55.734479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:55.734479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.734479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:26:55.734479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:26:55.734479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:26:55.734479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:26:55.734479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.734479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:55.734479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:55.734479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:55.734479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.734479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:55.734479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:55.734479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:55.734479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:55.734933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:55.734933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:55.734933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:55.734933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:55.734933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:55.734933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:55.734933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:55.734933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:55.734933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:55.734933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.734933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:55.740797 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:55.740947 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:55.741040 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:55.741185 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:55.741509 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:55.741716 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:55.741937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.742442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:55.742442 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:55.742442 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:55.742442 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:55.742442 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:55.742442 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:55.742442 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.742442 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:55.743130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.743130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:55.743130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:55.743130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:55.743130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.743130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:55.743130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:55.743130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.743130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:55.743130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:55.743130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:55.743130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.743588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:55.743588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:55.743588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:55.743588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:55.743588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:55.743588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:55.743588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:55.743588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:55.743588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:55.743588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:55.743588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:55.743588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:55.743588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:55.743971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.743971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:55.744243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.744243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:55.744243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:55.744243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:55.744243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:55.744243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:55.744243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.744243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:55.744243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:55.744243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.744243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:55.744243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:55.744243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.744243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:55.744243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:55.744243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:55.744878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:26:55.744878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:55.744878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.744878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:55.744878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:55.744878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:55.744878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:55.744878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:55.744878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:55.744878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:55.744878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:55.744878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:55.744878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:55.744878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:55.744878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:55.744878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:55.744878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:55.744878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:55.744878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:55.745448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:55.745448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:55.745448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:26:58.638818 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:26:58.639205 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:26:58.639205 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:26:58.639730 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:26:58.681353 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:26:58.682584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:58.683205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:26:58.683205 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:58.683205 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:26:58.683205 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:26:58.683205 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:58.683205 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:26:58.683205 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:58.683205 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:26:58.684195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:58.684195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:26:58.684195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:58.684195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:26:58.684195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:58.684195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:58.684195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:58.684195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:58.684195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:58.684195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:58.684195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:26:58.684195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:58.684635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:26:58.684635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:58.684635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:26:58.684635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:58.684635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:26:58.684635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:26:58.684635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:58.684635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:26:58.684635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:58.684635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:26:58.684635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:26:58.684635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:58.684635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:26:58.685019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:58.685019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:26:58.685019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:58.685019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:26:58.685019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:26:58.685019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:26:58.685019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:26:58.685019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:26:58.685019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:58.685019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:26:58.685019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:26:58.685019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:58.685019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:26:58.685019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:26:58.685019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:26:58.685474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:26:58.685474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:26:58.685474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:26:58.685474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:26:58.685474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:26:58.685474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:58.685474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:58.685474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:26:58.685474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:26:58.685474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:26:58.685474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:26:58.685474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:26:58.685474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:26:58.685474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:26:58.685474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:26:58.685474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:26:58.685474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:26:58.685474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:26:58.685474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:26:58.686283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:26:58.686283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:26:58.686283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:26:58.686283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:26:58.686283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:26:58.686283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:00.704196 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:00.704753 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:00.704753 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:00.707561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:00.707775 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:00.707903 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ba2b93c-4f6a-4134-a3a6-d65bc4e48afa, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:00.708128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:00.708317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:00.708473 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:00.708652 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:00.708795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.709316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:00.709316 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:00.709316 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:00.709316 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:00.709316 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:00.709316 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:00.709316 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.709316 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:00.710335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.710335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:00.710335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:00.710335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:00.710335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.710335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:00.710335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:00.710335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.710335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:00.710335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:00.710335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:00.710335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:00.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:00.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:00.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:00.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:00.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:00.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:00.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:00.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:00.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:00.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:00.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:00.710614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:00.710854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.710854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:00.711369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.711369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:00.711369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:00.711369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:00.711369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:00.711369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:00.711369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.711369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:00.711369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:00.711369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.711369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:00.711369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:00.711369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.711369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:00.711369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:00.711369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.711967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:00.711967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:00.711967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.711967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:00.711967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:00.711967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:00.711967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.711967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:00.711967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:00.711967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:00.711967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:00.711967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:00.711967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:00.711967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:00.711967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:00.711967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:00.711967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:00.711967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:00.711967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:00.712754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:00.712754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.712754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:00.717381 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:00.717578 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:00.717680 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:00.717801 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:27:00.718283 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:00.718462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.718905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:00.718905 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:00.718905 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:00.718905 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:00.718905 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:00.718905 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:00.718905 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.718905 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:00.719632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.719632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:00.719632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:00.719632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:00.719632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.719632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:00.719632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:00.719632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:00.719632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.719632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:00.719632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:00.719632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:00.719632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:00.719632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:00.719632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:00.719963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:00.719963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:00.719963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:00.719963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:00.719963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:00.719963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:00.719963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:00.719963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.719963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:00.720555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.720555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:00.720555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:00.720555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:00.720555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.720555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:00.720555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:00.720555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:27:00.720555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:00.720555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:00.720555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.720555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:00.720879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:00.720879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:00.720879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.720879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:00.720879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:00.720879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:00.720879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:00.720879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:00.720879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:00.720879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:00.720879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:00.720879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:00.720879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:00.720879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:00.721216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:00.721216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:00.721216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.721216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:00.721599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.721599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:00.721599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:00.721599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:00.721599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:00.721599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:00.721599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.721599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:00.721599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:00.721599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.721599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:00.721599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:00.721599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.721599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:00.721599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:00.721599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.722024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:00.722024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:00.722024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:27:00.722024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:00.722024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:00.722024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.722024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:00.722024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:00.722024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:00.722024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.722024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:00.722024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:00.722024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:00.722024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:00.722024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:00.722024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:00.722024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:00.722024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:00.722406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:00.722406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:00.722406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:00.722406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:00.722406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:00.722406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.722406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:00.728651 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:00.728871 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:00.729011 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:00.729123 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:00.729698 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:00.729949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.731400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:00.731400 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:00.731400 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:00.731400 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:00.731400 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:00.731400 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:00.731400 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.731400 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:00.732087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.732087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:00.732087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:00.732087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:00.732087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.732087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:00.732087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:00.732087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:00.732087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.732087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:00.732087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:00.732087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:00.732087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:00.732087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:00.732087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:00.732494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:00.732494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:00.732494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:00.732494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:00.732494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:00.732494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:00.732494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:00.732494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.732494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:00.732946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.732946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:00.732946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:00.732946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:00.732946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.732946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:00.732946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:00.732946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:00.732946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:00.732946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.732946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:00.733302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:00.733302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:00.733302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.733302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:00.733302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:00.733302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:00.733302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:00.733302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:00.733302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:00.733302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:00.733302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:00.733302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:00.733302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:00.733302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:00.733566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:00.733566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:00.733566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.733566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:00.733972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.733972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:00.733972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:00.733972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:00.733972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:00.733972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:00.733972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.733972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:00.733972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:00.733972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.733972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:00.733972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:00.733972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.733972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:00.733972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:00.733972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.733972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:00.734339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:00.734339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:00.734339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:00.734339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.734339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:00.734339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:00.734339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:00.734339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.734339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:00.734339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:00.734339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:00.734339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:00.734339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:00.734339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:00.734339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:00.734339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:00.734339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:00.734339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:00.734339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:00.734786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:00.734786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:00.734786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.734786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:00.740969 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:00.741185 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:00.741265 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:00.741397 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:00.742776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:00.743000 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:00.743232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.743745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:00.743745 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:00.743745 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:00.743745 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:00.743745 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:00.743745 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:00.743745 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.743745 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:00.744500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.744500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:00.744500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:00.744500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:00.744500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.744500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:00.744500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:00.744500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.744500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:00.744500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:00.744500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:00.744500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.744863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:00.744863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:00.744863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:00.744863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:00.744863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:00.744863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:00.744863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:00.744863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:00.744863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:00.744863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:00.744863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:00.744863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:00.744863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:00.745118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.745118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:00.745179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.745179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:00.745179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:00.745179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:00.745179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:00.745179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:00.745179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.745179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:00.745179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:00.745179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.745179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:00.745179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:00.745179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.745179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:00.745179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:00.745179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:00.745521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:00.745521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:00.745521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.745521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:00.745521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:00.745521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:00.745521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:00.745521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:00.745521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:00.745521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:00.745521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:00.745521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:00.745521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:00.745521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:00.745521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:00.745521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:00.745521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:00.745521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:00.745521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:00.745880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:00.745880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:00.745880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:03.639235 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:03.639640 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:03.639640 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:03.640222 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:03.677349 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:03.677523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:03.677982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:03.677982 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:03.677982 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:03.677982 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:03.677982 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:03.677982 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:03.677982 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:03.677982 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:03.678667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:03.678667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:03.678667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:03.678667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:03.678667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:03.678667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:03.678667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:03.678667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:03.678667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:03.678667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:03.678667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:03.678667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:03.678940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:03.678940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:03.678940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:03.678940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:03.678940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:03.678940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:03.678940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:03.678940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:03.678940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:03.678940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:03.678940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:03.678940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:03.678940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:03.679223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:03.679223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:03.679343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:03.679343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:03.679343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:03.679343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:03.679343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:03.679343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:03.679343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:03.679343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:03.679343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:03.679343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:03.679343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:03.679343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:03.679343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:03.679343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:03.679343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:03.679343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:03.679664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:03.679664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:03.679664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:03.679664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:03.679664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:03.679664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:03.679664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:03.679664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:03.679664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:03.679664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:03.679664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:03.679664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:03.679664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:03.679664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:03.679664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:03.679664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:03.679664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:03.679664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:03.679664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:03.680198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:03.680198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:03.680198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:05.705008 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:05.706367 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:05.706367 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:05.708328 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:05.708510 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:05.708612 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ba2b93c-4f6a-4134-a3a6-d65bc4e48afa, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:05.708761 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:05.708858 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:05.708998 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:05.709109 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:05.709233 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:05.709369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.711834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:05.711834 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:05.711834 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:05.711834 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:05.711834 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:05.711834 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:05.711834 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.711834 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:05.711834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.711834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:05.711834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:05.711834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:05.711834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.711834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:05.711834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:05.712426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.712426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:05.712426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:05.712426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:05.712426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.712426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:05.712426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:05.712426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:05.712426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:05.712426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:05.712426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:05.712426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:05.712426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:05.712426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:05.712679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:05.712679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:05.712679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:05.712679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:05.712679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.712679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:05.712679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.712679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:05.712679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:05.712679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:05.712679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:05.712679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:05.712679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.712679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:05.712981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:05.713441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:05.713441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:05.713441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:05.713441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:05.713441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:05.713441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:05.713441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:05.713441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:05.713441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.713441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:05.717651 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:05.718122 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:05.718302 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:05.718460 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:27:05.719648 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:05.721571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.723906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:05.723906 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:05.723906 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:05.723906 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:05.723906 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:05.723906 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:05.723906 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.723906 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:05.723906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.723906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:05.723906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:05.723906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:05.723906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.723906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:05.723906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:05.723906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:05.723906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.723906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:05.724315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:05.724315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:05.724315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:05.724315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:05.724315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:05.724315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:05.724315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:05.724315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:05.724315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:05.724315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:05.724315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:05.724315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:05.724315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.724315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:05.724315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.724804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:05.724804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:05.724804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:05.724804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.724804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:05.724804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:05.724804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:27:05.724804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:05.724804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:05.724804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.724804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:05.724804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:05.724804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:05.725017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.725017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:05.725017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:05.725017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:05.725017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:05.725017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:05.725017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:05.725017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:05.725017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:05.725017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:05.725017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:05.725017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:05.725017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:05.725017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:05.725273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.725273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:05.725273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.725273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:05.725273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:05.725273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:05.725273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:05.725273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:05.725273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.725273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:05.725273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:05.725273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.725273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:05.725273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:05.725273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.725533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:05.725533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:05.725533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.725533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:05.725533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:05.725533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:27:05.725533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:05.725533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:05.725533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.725533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:05.725533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:05.725533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:05.725533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.725533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:05.725533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:05.725533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:05.725533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:05.725533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:05.725533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:05.725992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:05.725992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:05.725992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:05.725992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:05.725992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:05.725992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:05.725992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:05.725992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.725992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:05.729150 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:05.729290 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:05.729442 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:05.729603 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:05.730101 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:05.730297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.731371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:05.731371 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:05.731371 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:05.731371 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:05.731371 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:05.731371 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:05.731371 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.731371 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:05.731989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.731989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:05.731989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:05.731989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:05.731989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.731989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:05.731989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:05.731989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:05.731989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.731989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:05.731989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:05.731989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:05.731989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:05.731989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:05.731989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:05.732316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:05.732316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:05.732316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:05.732316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:05.732316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:05.732316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:05.732316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:05.732316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.732316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:05.733134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.733134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:05.733134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:05.733134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:05.733134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.733134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:05.733134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:05.733134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:05.733134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:05.733134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.733134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:05.733399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:05.733399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:05.733399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.733399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:05.733399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:05.733399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:05.733399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:05.733399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:05.733399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:05.733399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:05.733399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:05.733399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:05.733399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:05.733399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:05.733650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:05.733650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:05.733650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.733650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:05.735040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.735040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:05.735040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:05.735040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:05.735040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:05.735040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:05.735040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.735040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:05.735040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:05.735040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.735040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:05.735040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:05.735040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.735040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:05.735040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:05.735040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.735040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:05.735402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:05.735402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:05.735402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:05.735402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.735402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:05.735402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:05.735402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:05.735402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.735402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:05.735402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:05.735402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:05.735402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:05.735402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:05.735402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:05.735402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:05.735402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:05.735402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:05.735402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:05.735402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:05.735878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:05.735878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:05.735878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.735878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:05.741217 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:05.741462 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:05.741788 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:05.741788 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:05.742116 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:05.742330 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:05.742571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.743001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:05.743001 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:05.743001 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:05.743001 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:05.743001 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:05.743001 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:05.743001 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.743001 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:05.744746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.744746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:05.744746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:05.744746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:05.744746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.744746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:05.744746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:05.744746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.744746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:05.744746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:05.744746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:05.744746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.745172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:05.745172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:05.745172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:05.745172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:05.745172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:05.745172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:05.745172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:05.745172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:05.745172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:05.745172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:05.745172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:05.745172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:05.745172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:05.745405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.745405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:05.745405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.745405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:05.745405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:05.745405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:05.745405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:05.745405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:05.745405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.745405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:05.745405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:05.745405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.745405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:05.745405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:05.745405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.745689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:05.745689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:05.745689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:05.745689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:05.745689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:05.745689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.745689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:05.745689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:05.745689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:05.745689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:05.745689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:05.745689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:05.745689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:05.745689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:05.745689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:05.745689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:05.745689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:05.745689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:05.745689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:05.746019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:05.746019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:05.746019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:05.746019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:05.746019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:05.746019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:08.639638 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:08.640330 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:08.640330 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:08.640486 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:08.668635 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:08.668924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:08.669529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:08.669529 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:08.669529 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:08.669529 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:08.669529 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:08.669529 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:08.669529 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:08.669529 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:08.670228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:08.670228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:08.670228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:08.670228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:08.670228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:08.670228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:08.670228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:08.670228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:08.670228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:08.670228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:08.670228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:08.670228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:08.670597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:08.670597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:08.670597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:08.670597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:08.670597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:08.670597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:08.670597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:08.670597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:08.670597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:08.670597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:08.670597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:08.670597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:08.670597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:08.670927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:08.670927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:08.670927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:08.670927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:08.670927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:08.670927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:08.670927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:08.670927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:08.670927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:08.670927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:08.670927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:08.670927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:08.670927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:08.670927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:08.670927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:08.671374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:08.671374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:08.671374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:08.671374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:08.671374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:08.671374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:08.671374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:08.671374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:08.671374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:08.671374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:08.671374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:08.671374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:08.671374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:08.671374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:08.671374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:08.671374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:08.671374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:08.671374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:08.671374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:08.671870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:08.671870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:08.671870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:08.671870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:08.671870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:08.671870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:10.707050 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:10.707773 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:10.707928 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:10.711294 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:10.711294 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:10.711294 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ba2b93c-4f6a-4134-a3a6-d65bc4e48afa, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:10.711294 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:10.711294 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:10.711511 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:10.711511 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:10.711511 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:10.711511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.711742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:10.711742 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:10.711742 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:10.711742 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:10.711742 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:10.711742 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:10.711742 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.711742 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:10.712886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.712886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:10.712886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:10.712886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:10.712886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.712886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:10.712886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:10.712886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.712886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:10.712886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:10.712886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:10.712886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.713178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:10.713178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:10.713178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:10.713178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:10.713178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:10.713178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:10.713178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:10.713178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:10.713178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:10.713178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:10.713178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:10.713178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:10.713178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:10.713467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.713467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:10.714937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.714937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:10.714937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:10.714937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:10.714937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:10.714937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:10.714937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.714937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:10.714937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:10.714937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.714937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:10.714937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:10.714937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.714937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:10.714937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:10.714937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.715323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:10.715323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:10.715323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.715323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:10.715323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:10.715323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:10.715323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.715323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:10.715323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:10.715323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:10.715323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:10.715323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:10.715323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:10.715323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:10.715323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:10.715323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:10.715323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:10.715323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:10.715323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:10.715711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:10.715711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.715711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:10.718173 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:10.718410 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:10.718547 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:10.718672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:27:10.718994 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:10.719219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.719649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:10.719649 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:10.719649 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:10.719649 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:10.719649 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:10.719649 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:10.719649 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.719649 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:10.720148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.720148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:10.720148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:10.720148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:10.720148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.720148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:10.720148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:10.720148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:10.720148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.720148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:10.720148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:10.720148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:10.720148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:10.720148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:10.720148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:10.720488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:10.720488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:10.720488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:10.720488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:10.720488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:10.720488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:10.720488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:10.720488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.720488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:10.720690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.720690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:10.720690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:10.720690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:10.720690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.720690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:10.720690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:10.720690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:27:10.720690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:10.720690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:10.720690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.720690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:10.720934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:10.720934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:10.720934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.720934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:10.720934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:10.720934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:10.720934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:10.720934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:10.720934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:10.720934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:10.720934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:10.720934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:10.720934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:10.720934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:10.721192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:10.721192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:10.721192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.721192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:10.721380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.721380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:10.721380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:10.721380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:10.721380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:10.721380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:10.721380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.721380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:10.721380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:10.721380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.721380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:10.721380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:10.721380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.721380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:10.721380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:10.721380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.721670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:10.721670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:10.721670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:27:10.721670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:10.721670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:10.721670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.721670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:10.721670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:10.721670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:10.721670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.721670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:10.721670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:10.721670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:10.721670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:10.721670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:10.721670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:10.721670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:10.721670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:10.722000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:10.722000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:10.722000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:10.722000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:10.722000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:10.722000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.722000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:10.729290 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:10.730109 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:10.730418 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:10.731126 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:10.731882 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:10.731882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.731955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:10.731955 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:10.731955 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:10.731955 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:10.731955 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:10.731955 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:10.731955 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.731955 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:10.732519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.732519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:10.732519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:10.732519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:10.732519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.732519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:10.732519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:10.732519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:10.732519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.732519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:10.732519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:10.732519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:10.732519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:10.732519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:10.732519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:10.732843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:10.732843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:10.732843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:10.732843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:10.732843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:10.732843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:10.732843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:10.732843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.732843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:10.733319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.733319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:10.733319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:10.733319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:10.733319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.733319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:10.733319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:10.733319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:10.733319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:10.733319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.733319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:10.733547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:10.733547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:10.733547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.733547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:10.733547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:10.733547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:10.733547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:10.733547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:10.733547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:10.733547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:10.733547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:10.733547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:10.733547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:10.733547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:10.733788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:10.733788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:10.733788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.733788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:10.734530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.734530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:10.734530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:10.734530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:10.734530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:10.734530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:10.734530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.734530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:10.734530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:10.734530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.734530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:10.734530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:10.734530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.734530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:10.734530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:10.734530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.734530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:10.734851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:10.734851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:10.734851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:10.734851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.734851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:10.734851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:10.734851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:10.734851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.734851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:10.734851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:10.734851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:10.734851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:10.734851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:10.734851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:10.734851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:10.734851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:10.734851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:10.734851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:10.734851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:10.735272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:10.735272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:10.735272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.735272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:10.742537 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:10.742537 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:10.742537 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:10.742537 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:10.742878 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:10.743594 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:10.744512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.745296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:10.745296 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:10.745296 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:10.745296 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:10.745296 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:10.745296 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:10.745296 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.745296 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:10.745891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.745891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:10.745891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:10.745891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:10.745891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.745891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:10.745891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:10.745891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.745891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:10.745891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:10.745891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:10.745891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.746198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:10.746198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:10.746198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:10.746198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:10.746198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:10.746198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:10.746198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:10.746198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:10.746198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:10.746198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:10.746198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:10.746198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:10.746198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:10.746503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.746503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:10.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:10.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:10.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:10.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:10.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:10.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:10.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:10.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:10.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:10.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:10.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:10.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:10.747136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:10.747136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:10.747136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.747136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:10.747136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:10.747136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:10.747136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:10.747136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:10.747136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:10.747136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:10.747136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:10.747136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:10.747136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:10.747136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:10.747136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:10.747136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:10.747136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:10.747136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:10.747136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:10.748588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:10.748588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:10.748588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:13.639963 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:13.640388 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:13.640388 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:13.640832 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:13.676579 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:13.676789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:13.677325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:13.677325 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:13.677325 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:13.677325 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:13.677325 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:13.677325 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:13.677325 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:13.677325 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:13.678394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:13.678394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:13.678394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:13.678394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:13.678394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:13.678394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:13.678394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:13.678394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:13.678394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:13.678394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:13.678394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:13.678394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:13.678665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:13.678665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:13.678665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:13.678665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:13.678665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:13.678665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:13.678665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:13.678665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:13.678665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:13.678665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:13.678665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:13.678665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:13.678665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:13.678881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:13.678881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:13.678881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:13.678881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:13.678881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:13.678881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:13.678881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:13.678881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:13.678881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:13.678881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:13.678881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:13.678881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:13.678881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:13.678881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:13.678881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:13.679206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:13.679206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:13.679206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:13.679206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:13.679206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:13.679206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:13.679206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:13.679206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:13.679206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:13.679206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:13.679206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:13.679206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:13.679206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:13.679206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:13.679206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:13.679206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:13.679206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:13.679206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:13.679206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:13.679539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:13.679539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:13.679539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:13.679539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:13.679539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:13.679539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:15.707931 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:15.708480 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:15.708480 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:15.709286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:15.710471 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:15.710747 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ba2b93c-4f6a-4134-a3a6-d65bc4e48afa, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:15.711084 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:15.711334 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:15.713474 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:15.713724 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:15.713969 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:15.714225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.715503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:15.715503 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:15.715503 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:15.715503 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:15.715503 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:15.715503 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:15.715503 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.715503 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:15.716369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.716369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:15.716369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:15.716369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:15.716369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.716369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:15.716369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:15.716369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.716369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:15.716369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:15.716369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:15.716369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.716802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:15.716802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:15.716802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:15.716802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:15.716802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:15.716802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:15.716802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:15.716802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:15.716802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:15.716802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:15.716802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:15.716802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:15.716802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:15.717228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.717228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:15.717228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.717228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:15.717228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:15.717228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:15.717228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:15.717228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:15.717228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.717228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:15.717228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:15.717228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.717228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:15.717228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:15.717228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.718229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:15.718229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:15.718229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.718229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:15.718229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:15.718229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.718229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:15.718229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:15.718229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:15.718229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.718229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:15.718229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:15.718229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:15.718229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:15.718229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:15.718229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:15.718229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:15.718229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:15.718229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:15.718821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:15.718821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:15.718821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:15.718821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:15.718821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.718821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:15.724964 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:15.724964 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:15.724964 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:15.724964 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:27:15.725519 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:15.725804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.726432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:15.726432 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:15.726432 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:15.726432 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:15.726432 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:15.726432 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:15.726432 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.726432 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:15.727176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.727176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:15.727176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:15.727176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:15.727176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.727176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:15.727176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:15.727176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:15.727176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.727176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:15.727176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:15.727176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:15.727176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:15.727176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:15.727176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:15.727667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:15.727667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:15.727667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:15.727667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:15.727667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:15.727667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:15.727667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:15.727667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.727667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:15.728090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.728090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:15.728090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:15.728090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:15.728090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.728090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:15.728090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:15.728090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:27:15.728090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:15.728090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:15.728090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.728090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:15.728483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:15.728483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:15.728483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.728483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:15.728483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:15.728483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:15.728483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:15.728483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:15.728483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:15.728483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:15.728483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:15.728483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:15.728483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:15.728483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:15.728889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:15.728889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:15.728889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.728889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:15.729201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.729201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:15.729201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:15.729201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:15.729201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:15.729201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:15.729201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.729201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:15.729201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:15.729201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.729201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:15.729201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:15.729201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.729201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:15.729201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:15.729201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.729771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:15.729771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:15.729771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:27:15.729771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:15.729771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:15.729771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.729771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:15.729771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:15.729771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:15.729771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.729771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:15.729771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:15.729771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:15.729771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:15.729771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:15.729771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:15.729771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:15.729771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:15.730350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:15.730350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:15.730350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:15.730350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:15.730350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:15.730350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.730350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:15.731312 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:15.731511 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:15.731628 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:15.731765 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:15.732540 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:15.732540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.732671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:15.732671 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:15.732671 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:15.732671 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:15.732671 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:15.732671 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:15.732671 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.732671 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:15.733332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.733332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:15.733332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:15.733332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:15.733332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.733332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:15.733332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:15.733332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:15.733332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.733332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:15.733332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:15.733332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:15.733332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:15.733332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:15.733332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:15.733750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:15.733750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:15.733750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:15.733750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:15.733750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:15.733750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:15.733750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:15.733750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.733750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:15.734195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.734195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:15.734195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:15.734195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:15.734195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.734195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:15.734195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:15.734195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:15.734195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:15.734195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.734195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:15.734511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:15.734511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:15.734511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.734511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:15.734511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:15.734511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:15.734511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:15.734511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:15.734511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:15.734511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:15.734511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:15.734511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:15.734511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:15.734511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:15.734912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:15.734912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:15.734912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.734912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:15.735312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.735312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:15.735312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:15.735312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:15.735312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:15.735312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:15.735312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.735312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:15.735312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:15.735312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.735312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:15.735312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:15.735312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.735312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:15.735312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:15.735312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.735312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:15.735672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:15.735672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:15.735672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:15.735672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.735672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:15.735672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:15.735672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:15.735672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.735672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:15.735672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:15.735672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:15.735672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:15.735672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:15.735672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:15.735672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:15.735672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:15.735672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:15.735672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:15.735672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:15.735979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:15.735979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:15.735979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.735979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:15.741971 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:15.742197 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:15.742335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:15.742508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:15.743435 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:15.743435 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:15.743435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.743568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:15.743568 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:15.743568 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:15.743568 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:15.743568 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:15.743568 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:15.743568 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.743568 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:15.744271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.744271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:15.744271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:15.744271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:15.744271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.744271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:15.744271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:15.744271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.744271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:15.744271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:15.744271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:15.744271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.744651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:15.744651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:15.744651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:15.744651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:15.744651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:15.744651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:15.744651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:15.744651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:15.744651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:15.744651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:15.744651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:15.744651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:15.744651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:15.745040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.745040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:15.746298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.746298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:15.746298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:15.746298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:15.746298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:15.746298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:15.746298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.746298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:15.746298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:15.746298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.746298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:15.746298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:15.746298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.746298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:15.746298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:15.746298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:15.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:15.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:15.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:15.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:15.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:15.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:15.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:15.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:15.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:15.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:15.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:15.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:15.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:15.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:15.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:15.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:15.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:15.746787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:15.747114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:15.747114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:15.747114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:18.640347 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:18.641385 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:18.641385 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:18.641385 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:18.677634 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:18.677763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:18.678173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:18.678173 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:18.678173 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:18.678173 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:18.678173 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:18.678173 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:18.678173 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:18.678173 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:18.678840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:18.678840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:18.678840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:18.678840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:18.678840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:18.678840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:18.678840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:18.678840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:18.678840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:18.678840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:18.678840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:18.678840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:18.679177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:18.679177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:18.679177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:18.679177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:18.679177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:18.679177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:18.679177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:18.679177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:18.679177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:18.679177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:18.679177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:18.679177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:18.679177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:18.679549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:18.679549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:18.679621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:18.679621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:18.679621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:18.679621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:18.679621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:18.679621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:18.679621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:18.679621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:18.679621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:18.679621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:18.679621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:18.679621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:18.679621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:18.679621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:18.679621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:18.679621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:18.680086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:18.680086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:18.680086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:18.680086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:18.680086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:18.680086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:18.680086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:18.680086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:18.680086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:18.680086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:18.680086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:18.680086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:18.680086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:18.680086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:18.680086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:18.680086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:18.680086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:18.680086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:18.680086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:18.680615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:18.680615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:18.680615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:20.708350 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:20.708932 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:20.708932 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:20.709609 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:20.709784 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:20.709939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ba2b93c-4f6a-4134-a3a6-d65bc4e48afa, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:20.710094 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:20.710238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:20.710487 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:20.710621 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:20.710773 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:20.710941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.711427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:20.711427 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:20.711427 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:20.711427 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:20.711427 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:20.711427 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:20.711427 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.711427 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:20.712102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.712102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:20.712102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:20.712102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:20.712102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.712102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:20.712102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:20.712102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.712102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:20.712102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:20.712102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:20.712102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.712432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:20.712432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:20.712432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:20.712432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:20.712432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:20.712432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:20.712432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:20.712432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:20.712432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:20.712432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:20.712432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:20.712432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:20.712432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:20.712664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.712664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:20.712954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.712954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:20.712954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:20.712954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:20.712954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:20.712954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:20.712954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.712954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:20.712954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:20.712954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.712954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:20.712954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:20.712954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.712954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:20.712954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:20.712954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.713341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:20.713341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:20.713341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.713341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:20.713341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:20.713341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:20.713341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.713341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:20.713341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:20.713341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:20.713341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:20.713341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:20.713341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:20.713341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:20.713341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:20.713341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:20.713341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:20.713341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:20.713341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:20.713700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:20.713700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.713700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:20.722724 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:20.722919 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:20.723017 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:20.723119 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:27:20.723450 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:20.723651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.724229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:20.724229 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:20.724229 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:20.724229 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:20.724229 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:20.724229 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:20.724229 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.724229 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:20.724722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.724722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:20.724722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:20.724722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:20.724722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.724722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:20.724722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:20.724722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:20.724722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.724722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:20.724722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:20.724722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:20.724722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:20.724722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:20.724722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:20.725027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:20.725027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:20.725027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:20.725027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:20.725027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:20.725027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:20.725027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:20.725027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.725027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:20.725302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.725302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:20.725302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:20.725302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:20.725302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.725302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:20.725302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:20.725302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:27:20.725302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:20.725302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:20.725302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.725302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:20.725630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:20.725630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:20.725630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.725630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:20.725630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:20.725630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:20.725630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:20.725630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:20.725630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:20.725630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:20.725630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:20.725630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:20.725630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:20.725630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:20.725938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:20.725938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:20.725938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.725938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:20.726032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.726032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:20.726032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:20.726032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:20.726032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:20.726032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:20.726032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.726032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:20.726032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:20.726032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.726032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:20.726032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:20.726032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.726032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:20.726032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:20.726032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.726443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:20.726443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:20.726443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:27:20.726443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:20.726443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:20.726443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.726443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:20.726443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:20.726443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:20.726443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.726443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:20.726443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:20.726443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:20.726443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:20.726443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:20.726443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:20.726443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:20.726443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:20.726809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:20.726809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:20.726809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:20.726809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:20.726809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:20.726809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.726809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:20.731604 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:20.731749 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:20.731830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:20.731948 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:20.732236 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:20.732433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.732787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:20.732787 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:20.732787 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:20.732787 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:20.732787 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:20.732787 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:20.732787 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.732787 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:20.733324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.733324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:20.733324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:20.733324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:20.733324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.733324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:20.733324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:20.733324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:20.733324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.733324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:20.733324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:20.733324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:20.733324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:20.733324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:20.733324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:20.733600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:20.733600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:20.733600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:20.733600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:20.733600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:20.733600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:20.733600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:20.733600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.733600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:20.733892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.733892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:20.733892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:20.733892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:20.733892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.733892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:20.733892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:20.733892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:20.733892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:20.733892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.733892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:20.734226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:20.734226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:20.734226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.734226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:20.734226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:20.734226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:20.734226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:20.734226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:20.734226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:20.734226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:20.734226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:20.734226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:20.734226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:20.734226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:20.734486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:20.734486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:20.734486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.734486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:20.734616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.734616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:20.734616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:20.734616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:20.734616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:20.734616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:20.734616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.734616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:20.734616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:20.734616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.734616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:20.734616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:20.734616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.734616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:20.734616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:20.734616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.734616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:20.734939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:20.734939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:20.734939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:20.734939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.734939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:20.734939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:20.734939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:20.734939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.734939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:20.734939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:20.734939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:20.734939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:20.734939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:20.734939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:20.734939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:20.734939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:20.734939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:20.734939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:20.734939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:20.735324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:20.735324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:20.735324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.735324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:20.742432 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:20.742778 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:20.742911 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:20.743232 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:20.743581 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:20.743757 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:20.743997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.744449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:20.744449 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:20.744449 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:20.744449 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:20.744449 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:20.744449 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:20.744449 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.744449 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:20.745655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.745655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:20.745655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:20.745655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:20.745655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.745655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:20.745655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:20.745655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.745655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:20.745655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:20.745655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:20.745655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.745917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:20.745917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:20.745917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:20.745917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:20.745917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:20.745917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:20.745917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:20.745917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:20.745917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:20.745917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:20.745917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:20.745917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:20.745917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:20.746181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.746181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:20.746935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.746935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:20.746935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:20.746935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:20.746935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:20.746935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:20.746935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.746935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:20.746935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:20.746935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.746935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:20.746935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:20.746935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.746935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:20.746935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:20.746935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:20.747327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:20.747327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:20.747327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.747327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:20.747327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:20.747327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:20.747327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:20.747327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:20.747327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:20.747327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:20.747327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:20.747327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:20.747327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:20.747327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:20.747327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:20.747327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:20.747327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:20.747327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:20.747327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:20.747620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:20.747620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:20.747620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:23.640709 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:23.641139 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:23.641139 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:23.641711 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:23.676441 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:23.676717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:23.677442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:23.677442 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:23.677442 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:23.677442 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:23.677442 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:23.677442 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:23.677442 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:23.677442 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:23.678299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:23.678299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:23.678299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:23.678299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:23.678299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:23.678299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:23.678299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:23.678299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:23.678299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:23.678299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:23.678299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:23.678299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:23.678543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:23.678543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:23.678543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:23.678543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:23.678543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:23.678543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:23.678543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:23.678543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:23.678543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:23.678543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:23.678543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:23.678543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:23.678543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:23.678799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:23.678799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:23.679572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:23.679572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:23.679572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:23.679572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:23.679572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:23.679572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:23.679572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:23.679572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:23.679572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:23.679572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:23.679572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:23.679572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:23.679572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:23.679572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:23.679572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:23.679572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:23.680087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:23.680087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:23.680087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:23.680087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:23.680087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:23.680087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:23.680087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:23.680087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:23.680087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:23.680087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:23.680087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:23.680087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:23.680087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:23.680087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:23.680087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:23.680087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:23.680087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:23.680087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:23.680087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:23.680448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:23.680448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:23.680448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:25.709288 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:25.710302 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:25.710302 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:25.711565 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:25.711822 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55f4749b-3181-4ab1-8c76-c41325e4f742, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:25.712017 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ba2b93c-4f6a-4134-a3a6-d65bc4e48afa, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:25.712183 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:25.712335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:25.712492 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:25.712629 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:25.712731 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:25.712873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.713484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:25.713484 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:25.713484 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:25.713484 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:25.713484 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:25.713484 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:25.713484 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.713484 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:25.714263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.714263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:25.714263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:25.714263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:25.714263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.714263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:25.714263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:25.714263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.714263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:25.714263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:25.714263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:25.714263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.715086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:25.715086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:25.715086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:25.715086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:25.715086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:25.715086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:25.715086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:25.715086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:25.715086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:25.715086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:25.715086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:25.715086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:25.715086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:25.715504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.715504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:25.715504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.715504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:25.715504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:25.715504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:25.715504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:25.715504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:25.715504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.715504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:25.715504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:25.715504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.715504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:25.715504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:25.715504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.716332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:25.716332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:25.716332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.716332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:25.716332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:25.716332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.716332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:25.716332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:25.716332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:25.716332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.716332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:25.716332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:25.716332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:25.716332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:25.716332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:25.716332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:25.716332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:25.716332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:25.716332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:25.717360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:25.717360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:25.717360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:25.717360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:25.717360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.717360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:25.723336 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:25.724167 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:25.724644 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:25.725112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:27:25.725755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:25.725755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.726231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:25.726231 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:25.726231 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:25.726231 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:25.726231 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:25.726231 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:25.726231 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.726231 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:25.727141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.727141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:25.727141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:25.727141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:25.727141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.727141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:25.727141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:25.727141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:25.727141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.727141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:25.727141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:25.727141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:25.727141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:25.727141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:25.727141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:25.727669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:25.727669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:25.727669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:25.727669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:25.727669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:25.727669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:25.727669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:25.727669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.727669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:25.727669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.727669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:25.727669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:25.727669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:25.727669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.728119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:25.728119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:25.728119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:27:25.728119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:25.728119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:25.728119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.728119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:25.728119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:25.728119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:25.728119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.728119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:25.728119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:25.728119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:25.729030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:25.729030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:25.729030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:25.729030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:25.729030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:25.729030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:25.729030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:25.729030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:25.729030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:25.729030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:25.729030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.729030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:25.729344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.729344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:25.729344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:25.729344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:25.729344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:25.729344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:25.729344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.729344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:25.729344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:25.729344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.729344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:25.729344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:25.729344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.729344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:25.729344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:25.729344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.729722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:25.729722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:25.729722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:27:25.729722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:25.729722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:25.729722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.729722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:25.729722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:25.729722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:25.729722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.729722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:25.729722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:25.729722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:25.729722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:25.729722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:25.729722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:25.729722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:25.729722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:25.730140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:25.730140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:25.730140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:25.730140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:25.730140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:25.730140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.730140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:25.733475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:25.733475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:25.733475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:25.733475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:25.734017 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:25.734223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.735093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:25.735093 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:25.735093 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:25.735093 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:25.735093 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:25.735093 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:25.735093 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.735093 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:25.735702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.735702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:25.735702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:25.735702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:25.735702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.735702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:25.735702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:25.735702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:25.735702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.735702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:25.735702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:25.735702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:25.735702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:25.735702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:25.735702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:25.736508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:25.736508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:25.736508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:25.736508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:25.736508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:25.736508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:25.736508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:25.736508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.736508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:25.736508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.736508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:25.736508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:25.736508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:25.736508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.736744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:25.736744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:25.736744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:25.736744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:25.736744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.736744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:25.736744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:25.736744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:25.736744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.736744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:25.736744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:25.736744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:25.736744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:25.736956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:25.736956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:25.736956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:25.736956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:25.736956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:25.736956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:25.736956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:25.736956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:25.736956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:25.736956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.736956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:25.737141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.737141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:25.737141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:25.737141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:25.737141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:25.737141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:25.737141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.737141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:25.737141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:25.737141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.737141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:25.737141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:25.737141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.737141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:25.737141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:25.737141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.737141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:25.738563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:25.738563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:25.738563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:25.738563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.738563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:25.738563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:25.738563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:25.738563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.738563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:25.738563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:25.738563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:25.738563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:25.738563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:25.738563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:25.738563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:25.738563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:25.738563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:25.738563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:25.738563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:25.738857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:25.738857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:25.738857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.738857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:25.743164 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:25.745609 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:25.745712 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:25.746188 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:25.746188 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:25.746350 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:25.746649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.749999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:25.749999 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:25.749999 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:25.749999 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:25.749999 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:25.749999 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:25.749999 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.749999 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:25.750609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.750609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:25.750609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:25.750609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:25.750609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.750609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:25.750609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:25.750609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.750609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:25.750609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:25.750609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:25.750609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.750875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:25.750875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:25.750875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:25.750875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:25.750875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:25.750875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:25.750875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:25.750875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:25.750875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:25.750875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:25.750875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:25.750875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:25.750875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:25.751155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.751155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:25.751308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.751308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:25.751308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:25.751308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:25.751308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:25.751308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:25.751308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.751308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:25.751308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:25.751308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.751308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:25.751308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:25.751308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.751308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:25.751308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:25.751308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:25.751589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:25.751589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:25.751589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.751589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:25.751589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:25.751589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:25.751589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:25.751589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:25.751589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:25.751589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:25.751589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:25.751589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:25.751589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:25.751589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:25.751589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:25.751589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:25.751589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:25.751589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:25.751589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:25.751939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:25.751939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:25.751939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:28.641188 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:28.644006 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:28.644006 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:28.644006 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:28.699583 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:28.699583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:28.700152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:28.700152 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:28.700152 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:28.700152 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:28.700152 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:28.700152 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:28.700152 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:28.700152 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:28.701409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.059 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:28.701409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:28.701409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:28.701409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:28.701409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:28.701409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:28.701409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:28.701409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:28.701409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:28.701409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:28.701409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:28.701409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:28.702000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:28.702000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:28.702000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:28.702000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:28.702000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:28.702000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:28.702000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:28.702000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:28.702000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:28.702000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:28.702000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:28.702000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:28.702000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:28.702617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:28.702617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:28.703497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:28.703497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:28.703497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:28.703497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:28.703497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:28.703497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:28.703497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:28.703497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:28.703497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:28.703497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:28.703497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:28.703497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:28.703497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:28.703497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:28.703497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:28.703497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:28.704319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:28.704319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:28.704319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:28.704319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:28.704319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:28.704319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:28.704319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:28.704319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:28.704319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:28.704319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:28.704319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:28.704319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:28.704319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:28.704319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:28.704319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:28.704319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:28.704319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:28.704319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:28.704319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:28.705313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:28.705313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:28.705313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:30.711394 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:30.711839 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:30.711839 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:30.712986 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:30.713135 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ba2b93c-4f6a-4134-a3a6-d65bc4e48afa, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:30.713240 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:30.713353 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:30.713453 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:30.713589 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:30.713682 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:30.713774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.714217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:30.714217 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:30.714217 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:30.714217 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:30.714217 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:30.714217 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:30.714217 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.714217 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:30.714784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.714784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:30.714784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:30.714784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:30.714784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.714784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:30.714784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:30.714784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.714784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:30.714784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:30.714784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:30.714784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:30.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:30.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:30.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:30.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:30.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:30.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:30.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:30.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:30.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:30.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:30.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:30.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:30.715358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.715358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:30.715703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.715703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:30.715703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:30.715703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:30.715703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:30.715703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:30.715703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.715703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:30.715703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:30.715703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.715703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:30.715703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:30.715703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.715703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:30.715703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:30.715703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.716014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:30.716014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:30.716014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.716014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:30.716014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:30.716014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:30.716014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.716014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:30.716014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:30.716014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:30.716014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:30.716014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:30.716014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:30.716014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:30.716014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:30.716014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:30.716014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:30.716014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:30.716014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:30.716388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:30.716388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.716388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:30.723858 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:30.724018 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:30.724113 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:30.724208 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:27:30.724417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:30.724550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.724841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:30.724841 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:30.724841 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:30.724841 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:30.724841 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:30.724841 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:30.724841 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.724841 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:30.725336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.725336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:30.725336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:30.725336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:30.725336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.725336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:30.725336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:30.725336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:30.725336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.725336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:30.725336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:30.725336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:30.725336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:30.725336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:30.725336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:30.725605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:30.725605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:30.725605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:30.725605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:30.725605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:30.725605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:30.725605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:30.725605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.725605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:30.725993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.725993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:30.725993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:30.725993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:30.725993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.725993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:30.725993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:30.725993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:27:30.725993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:30.725993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:30.725993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.725993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:30.726537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:30.726537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:30.726537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.726537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:30.726537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:30.726537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:30.726537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:30.726537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:30.726537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:30.726537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:30.726537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:30.726537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:30.726537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:30.726537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:30.727013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:30.727013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:30.727013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.727013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:30.727013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.727013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:30.727013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:30.727013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:30.727013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:30.727013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:30.727013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.727013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:30.727013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:30.727013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.727278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:30.727278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:30.727278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.727278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:30.727278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:30.727278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.727278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:30.727278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:30.727278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:27:30.727278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:30.727278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:30.727278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.727278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:30.727278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:30.727278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:30.727278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.727278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:30.727278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:30.727278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:30.727585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:30.727585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:30.727585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:30.727585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:30.727585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:30.727585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:30.727585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:30.727585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:30.727585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:30.727585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:30.727585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.727585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:30.732367 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:30.732515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:30.732589 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:30.732693 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:30.733074 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:30.733219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.733601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:30.733601 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:30.733601 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:30.733601 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:30.733601 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:30.733601 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:30.733601 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.733601 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:30.734185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.734185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:30.734185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:30.734185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:30.734185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.734185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:30.734185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:30.734185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:30.734185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.734185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:30.734185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:30.734185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:30.734185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:30.734185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:30.734185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:30.734450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:30.734450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:30.734450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:30.734450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:30.734450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:30.734450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:30.734450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:30.734450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.734450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:30.734749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.734749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:30.734749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:30.734749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:30.734749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.734749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:30.734749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:30.734749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:30.734749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:30.734749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.734749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:30.735334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:30.735334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:30.735334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.735334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:30.735334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:30.735334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:30.735334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:30.735334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:30.735334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:30.735334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:30.735334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:30.735334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:30.735334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:30.735334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:30.735679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:30.735679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:30.735679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.735679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:30.735679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.735679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:30.735679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:30.735679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:30.735679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:30.735679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:30.735679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.735679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:30.735679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:30.735679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.735947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:30.735947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:30.735947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.735947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:30.735947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:30.735947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.735947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:30.735947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:30.735947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:30.735947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:30.735947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.735947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:30.735947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:30.735947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:30.735947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.735947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:30.735947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:30.735947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:30.735947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:30.736268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:30.736268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:30.736268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:30.736268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:30.736268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:30.736268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:30.736268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:30.736268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:30.736268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:30.736268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.736268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:30.744577 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:30.744577 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:30.744577 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:30.744577 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:30.744577 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:30.744577 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:30.744760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.745053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:30.745053 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:30.745053 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:30.745053 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:30.745053 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:30.745053 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:30.745053 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.745053 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:30.745833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.745833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:30.745833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:30.745833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:30.745833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.745833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:30.745833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:30.745833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.745833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:30.745833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:30.745833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:30.745833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.746098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:30.746098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:30.746098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:30.746098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:30.746098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:30.746098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:30.746098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:30.746098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:30.746098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:30.746098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:30.746098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:30.746098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:30.746098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:30.746331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.746331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:30.746549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.746549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:30.746549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:30.746549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:30.746549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:30.746549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:30.746549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.746549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:30.746549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:30.746549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.746549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:30.746549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:30.746549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.746549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:30.746549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:30.746549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:30.747272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:30.747272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:30.747272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.747272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:30.747272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:30.747272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:30.747272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:30.747272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:30.747272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:30.747272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:30.747272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:30.747272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:30.747272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:30.747272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:30.747272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:30.747272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:30.747272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:30.747272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:30.747272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:30.748438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:30.748438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:30.748438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:33.642023 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:33.642485 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:33.642485 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:33.642990 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:33.690035 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:33.690279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:33.690638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:33.690638 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:33.690638 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:33.690638 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:33.690638 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:33.690638 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:33.690638 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:33.690638 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:33.691393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:33.691393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:33.691393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:33.691393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:33.691393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:33.691393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:33.691393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:33.691393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:33.691393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:33.691393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:33.691393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:33.691393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:33.691680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:33.691680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:33.691680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:33.691680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:33.691680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:33.691680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:33.691680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:33.691680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:33.691680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:33.691680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:33.691680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:33.691680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:33.691680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:33.691901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:33.691901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:33.692056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:33.692056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:33.692056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:33.692056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:33.692056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:33.692056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:33.692056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:33.692056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:33.692056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:33.692056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:33.692056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:33.692056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:33.692056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:33.692056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:33.692056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:33.692056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:33.692393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:33.692393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:33.692393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:33.692393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:33.692393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:33.692393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:33.692393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:33.692393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:33.692393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:33.692393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:33.692393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:33.692393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:33.692393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:33.692393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:33.692393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:33.692393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:33.692393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:33.692393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:33.692393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:33.692922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:33.692922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:33.692922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:35.711829 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:35.713283 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:35.713283 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:35.713640 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:35.713865 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ba2b93c-4f6a-4134-a3a6-d65bc4e48afa, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:35.714033 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:35.714216 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:35.714399 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:35.714562 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:35.714725 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:35.714883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.715474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:35.715474 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:35.715474 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:35.715474 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:35.715474 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:35.715474 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:35.715474 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.715474 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:35.717445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.717445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:35.717445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:35.717445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:35.717445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.717445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:35.717445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:35.717445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.717445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:35.717445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:35.717445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:35.717445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.717935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:35.717935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:35.717935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:35.717935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:35.717935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:35.717935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:35.717935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:35.717935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:35.717935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:35.717935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:35.717935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:35.717935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:35.717935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:35.718148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.718148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:35.718148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.718148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:35.718148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:35.718148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:35.718148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:35.718148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:35.718148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.718148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:35.718148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:35.718148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.718148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:35.718148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:35.718148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.718563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:35.718563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:35.718563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.718563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:35.718563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:35.718563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.718563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:35.718563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:35.718563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:35.718563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.718563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:35.718563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:35.718563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:35.718563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:35.718563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:35.718563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:35.718563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:35.718563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:35.718563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:35.719366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:35.719366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:35.719366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:35.719366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:35.719366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.719366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:35.724157 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:35.724384 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:35.724552 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:35.724727 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:27:35.725205 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:35.725435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.726331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:35.726331 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:35.726331 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:35.726331 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:35.726331 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:35.726331 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:35.726331 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.726331 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:35.726740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.726740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:35.726740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:35.726740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:35.726740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.726740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:35.726740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:35.726740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:35.726740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.726740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:35.726740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:35.726740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:35.726740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:35.726740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:35.726740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:35.727034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:35.727034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:35.727034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:35.727034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:35.727034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:35.727034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:35.727034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:35.727034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.727034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:35.727735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.727735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:35.727735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:35.727735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:35.727735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.727735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:35.727735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:35.727735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:27:35.727735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:35.727735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:35.727735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.727735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:35.727998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:35.727998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:35.727998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.727998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:35.727998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:35.727998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:35.727998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:35.727998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:35.727998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:35.727998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:35.727998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:35.727998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:35.727998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:35.727998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:35.728332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:35.728332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:35.728332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.728332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:35.728980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.728980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:35.728980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:35.728980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:35.728980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:35.728980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:35.728980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.728980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:35.728980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:35.728980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.728980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:35.728980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:35.728980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.728980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:35.728980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:35.728980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.729314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:35.729314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:35.729314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:27:35.729314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:35.729314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:35.729314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.729314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:35.729314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:35.729314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:35.729314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.729314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:35.729314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:35.729314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:35.729314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:35.729314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:35.729314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:35.729314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:35.729314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:35.729733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:35.729733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:35.729733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:35.729733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:35.729733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:35.729733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.729733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:35.732621 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:35.732904 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:35.733037 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:35.733195 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:35.733570 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:35.733810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.734363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:35.734363 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:35.734363 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:35.734363 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:35.734363 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:35.734363 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:35.734363 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.734363 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:35.735272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.735272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:35.735272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:35.735272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:35.735272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.735272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:35.735272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:35.735272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:35.735272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.735272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:35.735272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:35.735272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:35.735272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:35.735272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:35.735272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:35.735514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:35.735514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:35.735514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:35.735514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:35.735514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:35.735514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:35.735514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:35.735514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.735514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:35.736239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.736239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:35.736239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:35.736239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:35.736239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.736239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:35.736239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:35.736239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:35.736239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:35.736239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.736239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:35.736432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:35.736432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:35.736432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.736432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:35.736432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:35.736432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:35.736432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:35.736432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:35.736432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:35.736432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:35.736432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:35.736432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:35.736432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:35.736432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:35.736653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:35.736653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:35.736653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.736653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:35.738362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.738362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:35.738362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:35.738362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:35.738362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:35.738362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:35.738362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.738362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:35.738362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:35.738362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.738362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:35.738362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:35.738362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.738362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:35.738362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:35.738362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.738362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:35.738654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:35.738654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:35.738654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:35.738654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.738654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:35.738654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:35.738654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:35.738654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.738654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:35.738654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:35.738654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:35.738654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:35.738654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:35.738654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:35.738654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:35.738654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:35.738654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:35.738654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:35.738654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:35.738989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:35.738989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:35.738989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.738989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:35.744036 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:35.744244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:35.744352 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:35.744477 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:35.744779 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:35.744995 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:35.745277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.745745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:35.745745 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:35.745745 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:35.745745 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:35.745745 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:35.745745 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:35.745745 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.745745 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:35.746353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.746353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:35.746353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:35.746353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:35.746353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.746353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:35.746353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:35.746353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.746353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:35.746353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:35.746353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:35.746353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.746560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:35.746560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:35.746560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:35.746560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:35.746560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:35.746560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:35.746560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:35.746560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:35.746560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:35.746560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:35.746560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:35.746560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:35.746560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:35.746789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.746789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:35.747000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.747000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:35.747000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:35.747000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:35.747000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:35.747000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:35.747000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.747000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:35.747000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:35.747000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.747000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:35.747000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:35.747000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.747000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:35.747000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:35.747000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:35.747269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:35.747269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:35.747269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.747269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:35.747269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:35.747269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:35.747269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:35.747269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:35.747269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:35.747269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:35.747269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:35.747269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:35.747269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:35.747269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:35.747269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:35.747269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:35.747269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:35.747269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:35.747269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:35.747568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:35.747568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:35.747568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:38.642428 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:38.643538 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:38.643667 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:38.644444 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:38.684685 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:38.684924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:38.686169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:38.686169 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:38.686169 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:38.686169 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:38.686169 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:38.686169 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:38.686169 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:38.686169 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:38.686456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:38.686456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:38.686456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:38.686456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:38.686456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:38.686456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:38.686456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:38.686456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:38.686456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:38.686456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:38.686456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:38.686456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:38.686782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:38.686782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:38.686782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:38.686782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:38.686782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:38.686782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:38.686782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:38.686782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:38.686782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:38.686782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:38.686782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:38.686782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:38.686782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:38.686991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:38.686991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:38.687848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:38.687848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:38.687848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:38.687848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:38.687848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:38.687848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:38.687848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:38.687848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:38.687848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:38.687848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:38.687848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:38.687848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:38.687848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:38.687848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:38.687848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:38.687848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:38.688195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:38.688195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:38.688195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:38.688195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:38.688195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:38.688195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:38.688195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:38.688195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:38.688195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:38.688195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:38.688195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:38.688195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:38.688195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:38.688195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:38.688195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:38.688195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:38.688195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:38.688195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:38.688195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:38.688509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:38.688509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:38.688509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:40.712690 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:40.713089 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:40.713089 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:40.714177 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:40.714365 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ba2b93c-4f6a-4134-a3a6-d65bc4e48afa, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:40.714527 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:40.714648 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:40.714752 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:40.714884 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:40.714983 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-085e627c-39dc-43bf-a9ad-97ea27f2197b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:40.715112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.715621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:40.715621 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:40.715621 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:40.715621 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:40.715621 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:40.715621 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:40.715621 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.715621 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:40.716631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.716631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:40.716631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:40.716631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:40.716631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.716631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:40.716631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:40.716631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.716631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:40.716631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:40.716631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:40.716631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.716969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:40.716969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:40.716969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:40.716969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:40.716969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:40.716969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:40.716969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:40.716969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:40.716969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:40.716969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:40.716969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:40.716969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:40.716969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:40.717323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.717323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:40.717688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.717688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:40.717688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:40.717688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:40.717688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:40.717688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:40.717688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.717688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:40.717688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:40.717688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.717688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:40.717688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:40.717688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.717688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:40.717688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:40.717688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.718016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:40.718016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:40.718016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.718016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:40.718016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:40.718016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:40.718016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.718016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:40.718016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:40.718016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:40.718016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:40.718016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:40.718016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:40.718016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:40.718016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:40.718016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:40.718016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:40.718016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:40.718016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:40.718492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:40.718492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.718492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:40.724512 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:40.725711 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:40.725822 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:40.725944 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:27:40.726355 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:40.726529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.731498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:40.731498 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:40.731498 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:40.731498 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:40.731498 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:40.731498 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:40.731498 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.731498 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:40.732283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.732283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:40.732283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:40.732283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:40.732283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.732283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:40.732283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:40.732283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:40.732283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.732283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:40.732283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:40.732283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:40.732283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:40.732283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:40.732283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:40.732600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:40.732600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:40.732600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:40.732600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:40.732600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:40.732600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:40.732600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:40.732600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.732600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:40.733097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.733097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:40.733097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:40.733097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:40.733097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.733097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:40.733097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:40.733097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:27:40.733097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:40.733097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:40.733097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.733097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:40.733401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:40.733401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:40.733401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.733401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:40.733401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:40.733401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:40.733401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:40.733401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:40.733401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:40.733401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:40.733401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:40.733401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:40.733401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:40.733401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:40.733649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:40.733649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:40.733649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.733649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:40.734080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.734080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:40.734080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:40.734080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:40.734080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:40.734080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:40.734080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.734080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:40.734080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:40.734080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.734080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:40.734080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:40.734080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.734080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:40.734080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:40.734080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.734398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:40.734398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:40.734398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:27:40.734398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:40.734398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:40.734398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.734398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:40.734398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:40.734398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:40.734398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.734398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:40.734398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:40.734398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:40.734398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:40.734398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:40.734398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:40.734398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:40.734398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:40.734787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:40.734787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:40.734787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:40.734787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:40.734787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:40.734787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.734787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:40.738099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:40.738398 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:40.738504 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:40.738628 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:40.739755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:40.739974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.741016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:40.741016 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:40.741016 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:40.741016 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:40.741016 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:40.741016 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:40.741016 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.741016 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:40.742401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.742401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:40.742401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:40.742401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:40.742401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.742401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:40.742401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:40.742401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:40.742401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.742401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:40.742401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:40.742401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:40.742401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:40.742401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:40.742401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:40.742776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:40.742776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:40.742776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:40.742776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:40.742776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:40.742776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:40.742776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:40.742776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.742776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:40.742776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.742776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:40.742776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:40.742776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:40.742776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.743079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:40.743079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:40.743079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:40.743079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:40.743079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.743079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:40.743079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:40.743079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:40.743079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.743079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:40.743079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:40.743079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:40.743079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:40.743368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:40.743368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:40.743368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:40.743368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:40.743368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:40.743368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:40.743368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:40.743368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:40.743368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:40.743368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.743368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:40.743699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.743699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:40.743699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:40.743699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:40.743699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:40.743699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:40.743699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.743699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:40.743699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:40.743699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.743699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:40.743699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:40.743699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.743699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:40.743699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:40.743699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.743699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:40.744062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:40.744062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:40.744062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:40.744062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.744062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:40.744062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:40.744062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:40.744062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.744062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:40.744062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:40.744062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:40.744062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:40.744062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:40.744062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:40.744062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:40.744062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:40.744062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:40.744062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:40.744062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:40.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:40.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:40.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:40.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:40.744550 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:40.744587 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:40.748085 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:40.748085 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:40.748085 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:40.748085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.748085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:40.748085 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:40.748085 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:40.748085 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:40.748085 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:40.748085 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:40.748085 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.748085 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:40.748542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.748542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:40.748542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:40.748542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:40.748542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.748542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:40.748542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:40.748542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.748542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:40.748542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:40.748542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:40.748542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.748776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:40.748776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:40.748776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:40.748776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:40.748776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:40.748776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:40.748776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:40.748776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:40.748776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:40.748776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:40.748776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:40.748776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:40.748776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:40.749088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.749088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:40.749088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.749088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:40.749088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:40.749088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:40.749088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:40.749088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:40.749088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.749088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:40.749088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:40.749088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.749088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:40.749088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:40.749088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.749419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:40.749419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:40.749419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:40.749419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:40.749419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:40.749419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.749419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:40.749419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:40.749419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:40.749419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:40.749419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:40.749419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:40.749419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:40.749419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:40.749419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:40.749419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:40.749419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:40.749419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:40.749419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:40.749793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:40.749793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:40.749793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:40.749793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:40.749793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:40.749793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:43.643724 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:43.644201 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:43.644327 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:43.645343 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:43.700581 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:43.700581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:43.700990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:43.700990 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:43.700990 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:43.700990 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:43.700990 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:43.700990 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:43.700990 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:43.700990 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:43.701973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.057 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:43.701973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:43.701973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:43.701973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:43.701973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:43.701973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:43.701973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:43.701973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:43.701973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:43.701973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:43.701973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:43.701973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:43.702236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:43.702236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:43.702236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:43.702236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:43.702236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:43.702236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:43.702236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:43.702236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:43.702236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:43.702236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:43.702236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:43.702236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:43.702236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:43.702474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:43.702474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:43.703097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:43.703097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:43.703097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:43.703097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:43.703097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:43.703097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:43.703097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:43.703097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:43.703097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:43.703097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:43.703097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:43.703097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:43.703097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:43.703097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:43.703097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:43.703097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:43.703542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:43.703542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:43.703542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:43.703542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:43.703542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:43.703542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:43.703542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:43.703542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:43.703542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:43.703542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:43.703542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:43.703542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:43.703542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:43.703542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:43.703542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:43.703542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:43.703542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:43.703542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:43.703542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:43.703883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:43.703883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:43.703883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:45.713742 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:45.718462 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:45.721300 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:45.727498 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:45.728050 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:45.728362 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:45.728799 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:45.728799 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:45.728895 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:45.729014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.729688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:45.729688 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:45.729688 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:45.729688 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:45.729688 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:45.729688 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:45.729688 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.729688 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:45.730820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.011 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.730820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:45.730820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:45.730820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:45.730820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.730820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:45.730820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:45.730820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.730820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:45.730820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:45.730820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:45.730820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.731288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:45.731288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:45.731288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:45.731288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:45.731288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:45.731288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:45.731288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:45.731288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:45.731288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:45.731288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:45.731288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:45.731288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:45.731288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:45.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:45.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:45.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:45.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:45.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:45.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:45.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:45.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:45.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:45.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:45.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.732593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:45.732593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:45.732593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.732593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:45.732593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:45.732593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.732593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:45.732593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:45.732593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:45.732593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.732593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:45.732593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:45.732593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:45.732593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:45.732593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:45.732593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:45.732593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:45.732593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:45.732593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:45.733809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:45.733809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:45.733809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:45.733809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:45.733809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.733809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:45.733809 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:45.733809 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:45.733809 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:27:45.733809 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:45.734186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.734186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:45.734186 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:45.734186 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:45.734186 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:45.734186 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:45.734186 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:45.734186 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.734186 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:45.734610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.734610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:45.734610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:45.734610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:45.734610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.734610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:45.734610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:45.734610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:45.734610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.734610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:45.734610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:45.734610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:45.734610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:45.734610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:45.734610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:45.735077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:45.735077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:45.735077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:45.735077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:45.735077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:45.735077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:45.735077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:45.735077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.735077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:45.735542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.735542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:45.735542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:45.735542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:45.735542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.735542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:45.735542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:45.735542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:27:45.735542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:45.735542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:45.735542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.735542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:45.736218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:45.736218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:45.736218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.736218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:45.736218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:45.736218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:45.736218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:45.736218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:45.736218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:45.736218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:45.736218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:45.736218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:45.736218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:45.736218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:45.736689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:45.736689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:45.736689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.736689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:45.736689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.736689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:45.736689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:45.736689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:45.736689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:45.736689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:45.736689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.736689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:45.736689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:45.736689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.737034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:45.737034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:45.737034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.737034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:45.737034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:45.737034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.737034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:45.737034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:45.737034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:27:45.737034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:45.737034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:45.737034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.737034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:45.737034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:45.737034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:45.737034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.737034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:45.737034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:45.737034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:45.737404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:45.737404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:45.737404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:45.737404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:45.737404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:45.737404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:45.737404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:45.737404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:45.737404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:45.737404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:45.737404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.737404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:45.738445 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:45.738688 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:45.738820 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:45.738974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:45.739504 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:45.739755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.740356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:45.740356 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:45.740356 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:45.740356 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:45.740356 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:45.740356 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:45.740356 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.740356 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:45.741011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.741011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:45.741011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:45.741011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:45.741011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.741011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:45.741011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:45.741011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:45.741011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.741011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:45.741011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:45.741011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:45.741011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:45.741011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:45.741011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:45.742083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:45.742083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:45.742083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:45.742083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:45.742083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:45.742083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:45.742083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:45.742083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.742083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:45.742083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.742083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:45.742083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:45.742083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:45.742083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.742621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:45.742621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:45.742621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:45.742621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:45.742621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.742621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:45.742621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:45.742621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:45.742621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.742621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:45.742621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:45.742621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:45.742621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:45.742876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:45.742876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:45.742876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:45.742876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:45.742876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:45.742876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:45.742876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:45.742876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:45.742876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:45.742876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.742876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:45.743735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.743735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:45.743735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:45.743735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:45.743735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:45.743735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:45.743735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.743735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:45.743735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:45.743735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.743735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:45.743735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:45.743735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.743735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:45.743735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:45.743735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.743735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:45.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:45.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:45.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:45.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:45.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:45.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:45.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:45.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:45.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:45.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:45.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:45.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:45.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:45.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:45.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:45.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:45.744428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:45.745173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:45.745173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:45.745173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.745173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:45.745173 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:45.747939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:45.747939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:45.747939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:45.748112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:45.748424 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:45.748673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.751638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:45.751638 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:45.751638 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:45.751638 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:45.751638 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:45.751638 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:45.751638 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.751638 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:45.752481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.752481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:45.752481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:45.752481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:45.752481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.752481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:45.752481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:45.752481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.752481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:45.752481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:45.752481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:45.752481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.752865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:45.752865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:45.752865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:45.752865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:45.752865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:45.752865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:45.752865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:45.752865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:45.752865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:45.752865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:45.752865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:45.752865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:45.752865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:45.753198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.753198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:45.756593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.756593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:45.756593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:45.756593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:45.756593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:45.756593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:45.756593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.756593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:45.756593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:45.756593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.756593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:45.756593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:45.756593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.756593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:45.756593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:45.756593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:45.757091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:45.757091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:45.757091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.757091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:45.757091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:45.757091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:45.757091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:45.757091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:45.757091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:45.757091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:45.757091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:45.757091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:45.757091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:45.757091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:45.757091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:45.757091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:45.757091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:45.757091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:45.757091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:45.757740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:45.757740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:45.757740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:48.644244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:48.646219 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:48.646605 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:48.647776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:48.725317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:48.725317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:48.725569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:48.725569 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:48.725569 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:48.725569 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:48.725569 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:48.725569 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:48.725569 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:48.725569 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:48.726092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.079 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:48.726092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:48.726092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:48.726092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:48.726092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:48.726092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:48.726092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:48.726092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:48.726092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:48.726092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:48.726092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:48.726092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:48.726441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:48.726441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:48.726441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:48.726441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:48.726441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:48.726441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:48.726441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:48.726441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:48.726441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:48.726441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:48.726441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:48.726441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:48.726441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:48.726821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:48.726821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:48.726983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:48.726983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:48.726983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:48.726983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:48.726983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:48.726983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:48.726983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:48.726983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:48.726983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:48.726983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:48.726983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:48.726983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:48.726983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:48.726983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:48.726983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:48.726983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:48.727428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:48.727428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:48.727428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:48.727428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:48.727428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:48.727428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:48.727428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:48.727428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:48.727428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:48.727428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:48.727428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:48.727428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:48.727428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:48.727428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:48.727428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:48.727428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:48.727428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:48.727428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:48.727428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:48.727890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:48.727890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:48.727890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:50.715056 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:50.716272 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:50.716373 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:50.719571 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:50.720390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:50.720390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:50.720390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:50.720390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:50.720390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:50.720390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.721029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:50.721029 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:50.721029 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:50.721029 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:50.721029 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:50.721029 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:50.721029 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.721029 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:50.721929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.721929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:50.721929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:50.721929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:50.721929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.721929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:50.721929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:50.721929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.721929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:50.721929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:50.721929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:50.721929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.722216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:50.722216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:50.722216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:50.722216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:50.722216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:50.722216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:50.722216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:50.722216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:50.722216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:50.722216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:50.722216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:50.722216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:50.722216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:50.722602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.722602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:50.722664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.722664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:50.722664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:50.722664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:50.722664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:50.722664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:50.722664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.722664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:50.722664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:50.722664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.722664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:50.722664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:50.722664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.722664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:50.722664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:50.722664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.723198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:50.723198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:50.723198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.723198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:50.723198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:50.723198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:50.723198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.723198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:50.723198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:50.723198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:50.723198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:50.723198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:50.723198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:50.723198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:50.723198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:50.723198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:50.723198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:50.723198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:50.723198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:50.723902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:50.723902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.723902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:50.732174 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:50.732585 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:50.732693 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:50.732803 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:27:50.733521 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:50.733741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.734379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:50.734379 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:50.734379 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:50.734379 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:50.734379 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:50.734379 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:50.734379 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.734379 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:50.735318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.735318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:50.735318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:50.735318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:50.735318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.735318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:50.735318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:50.735318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:50.735318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.735318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:50.735318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:50.735318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:50.735318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:50.735318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:50.735318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:50.736213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:50.736213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:50.736213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:50.736213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:50.736213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:50.736213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:50.736213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:50.736213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.736213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:50.736213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.736213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:50.736213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:50.736213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:50.736213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.736610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:50.736610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:50.736610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:27:50.736610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:50.736610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:50.736610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.736610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:50.736610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:50.736610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:50.736610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.736610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:50.736610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:50.736610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:50.736924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:50.736924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:50.736924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:50.736924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:50.736924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:50.736924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:50.736924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:50.736924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:50.736924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:50.736924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:50.736924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.736924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:50.737135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.737135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:50.737135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:50.737135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:50.737135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:50.737135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:50.737135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.737135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:50.737135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:50.737135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.737135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:50.737135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:50.737135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.737135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:50.737135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:50.737135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.737484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:50.737484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:50.737484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:27:50.737484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:50.737484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:50.737484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.737484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:50.737484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:50.737484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:50.737484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.737484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:50.737484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:50.737484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:50.737484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:50.737484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:50.737484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:50.737484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:50.737484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:50.737804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:50.737804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:50.737804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:50.737804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:50.737804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:50.737804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.737804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:50.738854 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:50.738854 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:50.739092 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:50.739092 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:50.739436 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:50.739718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.742445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:50.742445 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:50.742445 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:50.742445 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:50.742445 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:50.742445 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:50.742445 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.742445 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:50.743047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.743047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:50.743047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:50.743047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:50.743047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.743047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:50.743047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:50.743047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:50.743047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.743047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:50.743047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:50.743047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:50.743047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:50.743047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:50.743047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:50.743602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:50.743602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:50.743602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:50.743602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:50.743602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:50.743602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:50.743602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:50.743602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.743602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:50.743966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.743966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:50.743966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:50.743966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:50.743966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.743966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:50.743966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:50.743966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:50.743966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:50.743966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.743966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:50.744407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:50.744407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:50.744407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.744407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:50.744407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:50.744407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:50.744407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:50.744407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:50.744407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:50.744407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:50.744407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:50.744407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:50.744407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:50.744407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:50.744955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:50.744955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:50.744955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.744955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:50.744955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.744955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:50.744955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:50.744955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:50.744955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:50.744955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:50.744955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.744955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:50.744955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:50.744955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.745595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:50.745595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:50.745595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.745595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:50.745595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:50.745595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.745595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:50.745595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:50.745595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:50.745595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:50.745595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.745595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:50.745595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:50.745595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:50.745595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.745595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:50.745595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:50.745595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:50.745595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:50.746958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:50.746958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:50.746958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:50.746958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:50.746958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:50.746958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:50.746958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:50.746958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:50.746958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:50.746958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.746958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:50.747643 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:50.747643 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:50.747643 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:50.747792 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:50.748783 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:50.748783 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:50.748783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.748906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:50.748906 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:50.748906 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:50.748906 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:50.748906 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:50.748906 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:50.748906 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.748906 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:50.749468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.749468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:50.749468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:50.749468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:50.749468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.749468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:50.749468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:50.749468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.749468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:50.749468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:50.749468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:50.749468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.749850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:50.749850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:50.749850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:50.749850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:50.749850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:50.749850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:50.749850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:50.749850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:50.749850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:50.749850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:50.749850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:50.749850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:50.749850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:50.750175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.750175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:50.750175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.750175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:50.750175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:50.750175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:50.750175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:50.750175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:50.750175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.750175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:50.750175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:50.750175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.750175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:50.750175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:50.750175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.750444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:50.750444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:50.750444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:50.750444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:50.750444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:50.750444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.750444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:50.750444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:50.750444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:50.750444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:50.750444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:50.750444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:50.750444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:50.750444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:50.750444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:50.750444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:50.750444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:50.750444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:50.750444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:50.750772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:50.750772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:50.750772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:50.750772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:50.750772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:50.750772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:53.646226 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:53.646619 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:53.646686 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:53.647348 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:53.670444 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=242 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:27:53.701804 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:53.701804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:53.702808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:53.702808 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:53.702808 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:53.702808 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:53.702808 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:53.702808 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:53.702808 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:53.702808 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:53.703117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:53.703117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:53.703117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:53.703117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:53.703117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:53.703117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:53.703117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:53.703117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:53.703117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:53.703117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:53.703117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:53.703117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:53.703503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:53.703503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:53.703503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:53.703503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:53.703503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:53.703503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:53.703503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:53.703503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:53.703503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:53.703503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:53.703503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:53.703503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:53.703503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:53.703855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:53.703855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:53.703855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:53.703855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:53.703855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:53.703855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:53.703855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:53.703855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:53.703855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:53.703855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:53.703855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:53.703855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:53.703855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:53.703855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:53.703855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:53.704318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:53.704318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:53.704318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:53.704318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:53.704318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:53.704318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:53.704318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:53.704318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:53.704318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:53.704318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:53.704318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:53.704318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:53.704318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:53.704318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:53.704318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:53.704318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:53.704318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:53.704318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:53.704318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:53.704810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:53.704810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:53.704810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:53.704810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:53.704810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:53.704810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:55.716357 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:55.717187 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:55.717335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:55.719133 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:55.719390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:55.719588 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:55.719720 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:55.719846 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:55.719976 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:55.720160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.720683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:55.720683 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:55.720683 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:55.720683 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:55.720683 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:55.720683 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:55.720683 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.720683 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:55.721671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.721671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:55.721671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:55.721671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:55.721671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.721671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:55.721671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:55.721671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.721671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:55.721671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:55.721671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:55.721671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.721966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:55.721966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:55.721966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:55.721966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:55.721966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:55.721966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:55.721966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:55.721966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:55.721966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:55.721966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:55.721966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:55.721966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:55.721966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:55.722223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.722223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:55.724128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.724128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:55.724128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:55.724128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:55.724128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:55.724128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:55.724128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.724128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:55.724128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:55.724128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.724128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:55.724128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:55.724128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.724128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:55.724128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:55.724128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.724610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:27:55.724610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:55.724610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.724610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:55.724610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:55.724610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:55.724610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.724610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:55.724610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:55.724610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:55.724610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:55.724610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:55.724610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:55.724610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:55.724610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:55.724610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:55.724610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:55.724610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:55.724610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:55.724968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:55.724968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.724968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:55.732763 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:55.733787 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:55.734028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:55.734285 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:27:55.735093 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:55.735334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.736663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:55.736663 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:55.736663 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:55.736663 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:55.736663 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:55.736663 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:55.736663 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.736663 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:55.737829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.737829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:55.737829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:55.737829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:55.737829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.737829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:55.737829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:55.737829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:55.737829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.737829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:55.737829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:55.737829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:55.737829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:55.737829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:55.737829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:55.738155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:55.738155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:55.738155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:55.738155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:55.738155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:55.738155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:55.738155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:55.738155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.738155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:55.739274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.739274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:55.739274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:55.739274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:55.739274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.739274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:55.739274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:55.739274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:27:55.739274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:55.739274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:55.739274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.739274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:55.739575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:55.739575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:55.739575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.739575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:55.739575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:55.739575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:55.739575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:55.739575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:55.739575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:55.739575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:55.739575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:55.739575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:55.739575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:55.739575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:55.739856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:55.739856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:55.739856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.739856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:55.741785 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:55.744474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.744474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:55.744474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:55.744474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:55.744474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:55.744474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:55.744474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.744474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:55.744474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:55.744474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.744474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:55.744474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:55.744474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.744474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:55.744474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:55.744474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.744852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:27:55.744852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:27:55.744852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:27:55.744852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:55.744852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:55.744852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.744852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:55.744852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:55.744852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:55.744852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.744852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:55.744852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:55.744852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:55.744852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:55.744852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:55.744852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:55.744852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:55.744852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:55.745243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:55.745243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:55.745243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:55.745243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:55.745243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:55.745243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.745243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:55.745243 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:55.745517 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:55.746008 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:55.746559 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:55.746776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.747298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:55.747298 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:55.747298 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:55.747298 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:55.747298 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:55.747298 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:55.747298 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.747298 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:55.748469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.748469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:27:55.748469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:55.748469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:55.748469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.748469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:55.748469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:55.748469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:27:55.748469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.748469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:27:55.748469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:55.748469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:27:55.748469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:55.748469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:27:55.748469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:27:55.748798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:55.748798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:27:55.748798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:55.748798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:27:55.748798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:27:55.748798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:55.748798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:27:55.748798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.748798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:27:55.750452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.750452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:55.750452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:55.750452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:55.750452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.750452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:55.750452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:55.750452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:55.750452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:55.750452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.750452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:55.750693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:55.750693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:55.750693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.750693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:55.750693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:55.750693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:55.750693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:55.750693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:55.750693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:55.750693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:55.750693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:55.750693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:55.750693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:55.750693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:55.750946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:55.750946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:55.750946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.750946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:55.753435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.753435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:55.753435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:55.753435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:55.753435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:55.753435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:55.753435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.753435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:55.753435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:55.753435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.753435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:55.753435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:55.753435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.753435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:55.753435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:55.753435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.753435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:27:55.753844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:27:55.753844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:27:55.753844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:27:55.753844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.753844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:55.753844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:55.753844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:55.753844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.753844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:55.753844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:55.753844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:55.753844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:55.753844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:55.753844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:55.753844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:55.753844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:55.753844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:55.753844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:55.753844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:55.754970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:55.754970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:55.754970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.754970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:55.754970 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:55.755429 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:55.755429 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:55.755799 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:55.756200 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:55.756460 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:55.756609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.757358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:55.757358 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:55.757358 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:55.757358 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:55.757358 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:55.757358 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:55.757358 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.757358 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:55.758969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.758969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:55.758969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:55.758969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:55.758969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.758969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:55.758969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:55.758969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.758969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:55.758969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:55.758969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:55.758969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.759410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:55.759410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:55.759410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:55.759410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:55.759410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:55.759410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:55.759410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:55.759410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:55.759410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:55.759410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:55.759410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:55.759410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:55.759410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:55.759672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.759672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:55.761054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.761054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:55.761054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:55.761054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:55.761054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:55.761054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:55.761054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.761054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:55.761054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:55.761054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.761054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:55.761054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:55.761054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.761054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:55.761054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:55.761054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:55.761399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:27:55.761399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:55.761399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.761399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:55.761399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:55.761399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:55.761399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:55.761399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:55.761399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:55.761399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:55.761399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:55.761399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:55.761399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:55.761399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:55.761399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:55.761399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:55.761399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:55.761399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:55.761399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:55.761907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:55.761907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:55.761907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:27:58.646697 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:27:58.647369 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:27:58.647369 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:27:58.647687 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:27:58.687384 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:27:58.687635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:58.688226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:27:58.688226 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:58.688226 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:27:58.688226 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:27:58.688226 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:58.688226 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:27:58.688226 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:58.688226 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:27:58.689442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:58.689442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:27:58.689442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:58.689442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:27:58.689442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:58.689442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:58.689442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:58.689442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:58.689442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:58.689442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:58.689442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:27:58.689442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:58.690298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:27:58.690298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:58.690298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:27:58.690298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:58.690298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:27:58.690298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:27:58.690298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:58.690298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:27:58.690298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:58.690298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:27:58.690298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:27:58.690298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:58.690298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:27:58.691114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:58.691114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:27:58.691114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:58.691114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:27:58.691114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:27:58.691114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:27:58.691114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:27:58.691114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:27:58.691114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:58.691114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:27:58.691114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:27:58.691114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:58.691114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:27:58.691114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:27:58.691114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:27:58.691942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:27:58.691942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:27:58.691942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:27:58.691942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:27:58.691942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:27:58.691942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:58.691942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:58.691942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:27:58.691942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:27:58.691942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:27:58.691942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:27:58.691942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:27:58.691942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:27:58.691942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:27:58.691942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:27:58.691942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:27:58.691942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:27:58.691942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:27:58.691942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:27:58.693359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:27:58.693359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:27:58.693359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:27:58.693359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:27:58.693359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:27:58.693359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:00.717233 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:00.718729 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:00.718729 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:00.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:00.720128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:00.720128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:00.720128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:00.720128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:00.720128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:00.721240 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:00.722121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.723858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:00.723858 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:00.723858 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:00.723858 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:00.723858 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:00.723858 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:00.723858 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.723858 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:00.724694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.724694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:00.724694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:00.724694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:00.724694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.724694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:00.724694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:00.724694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.724694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:00.724694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:00.724694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:00.724694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.725018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:00.725018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:00.725018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:00.725018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:00.725018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:00.725018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:00.725018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:00.725018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:00.725018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:00.725018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:00.725018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:00.725018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:00.725018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:00.725340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.725340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:00.725680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.725680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:00.725680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:00.725680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:00.725680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:00.725680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:00.725680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.725680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:00.725680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:00.725680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.725680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:00.725680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:00.725680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.725680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:00.725680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:00.725680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.726047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:00.726047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:00.726047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.726047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:00.726047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:00.726047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:00.726047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.726047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:00.726047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:00.726047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:00.726047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:00.726047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:00.726047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:00.726047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:00.726047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:00.726047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:00.726047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:00.726047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:00.726047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:00.726500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:00.726500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.726500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:00.733775 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:00.734117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:00.734201 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:00.734318 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:28:00.734759 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:00.736068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.736647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:00.736647 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:00.736647 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:00.736647 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:00.736647 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:00.736647 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:00.736647 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.736647 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:00.737215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.737215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:00.737215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:00.737215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:00.737215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.737215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:00.737215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:00.737215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:00.737215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.737215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:00.737215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:00.737215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:00.737215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:00.737215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:00.737215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:00.737859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:00.737859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:00.737859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:00.737859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:00.737859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:00.737859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:00.737859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:00.737859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.737859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:00.737859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.737859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:00.737859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:00.737859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:00.737859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.738154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:00.738154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:00.738154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:28:00.738154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:00.738154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:00.738154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.738154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:00.738154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:00.738154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:00.738154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.738154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:00.738154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:00.738154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:00.738797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:00.738797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:00.738797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:00.738797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:00.738797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:00.738797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:00.738797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:00.738797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:00.738797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:00.738797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:00.738797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.738797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:00.739311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.739311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:00.739311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:00.739311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:00.739311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:00.739311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:00.739311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.739311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:00.739311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:00.739311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.739311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:00.739311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:00.739311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.739311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:00.739311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:00.739311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.739616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:00.739616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:00.739616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:28:00.739616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:00.739616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:00.739616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.739616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:00.739616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:00.739616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:00.739616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.739616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:00.739616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:00.739616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:00.739616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:00.739616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:00.739616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:00.739616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:00.739616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:00.739956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:00.739956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:00.739956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:00.739956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:00.739956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:00.739956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.739956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:00.744726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:00.745050 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:00.745173 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:00.746455 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:00.746455 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:00.746455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.746931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:00.746931 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:00.746931 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:00.746931 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:00.746931 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:00.746931 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:00.746931 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.746931 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:00.747909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.747909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:00.747909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:00.747909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:00.747909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.747909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:00.747909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:00.747909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:00.747909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.747909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:00.747909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:00.747909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:00.747909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:00.747909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:00.747909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:00.748208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:00.748208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:00.748208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:00.748208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:00.748208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:00.748208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:00.748208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:00.748208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.748208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:00.748208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.748208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:00.748208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:00.748208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:00.748208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.748481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:00.748481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:00.748481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:00.748481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:00.748481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.748481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:00.748481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:00.748481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:00.748481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.748481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:00.748481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:00.748481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:00.748481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:00.748737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:00.748737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:00.748737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:00.748737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:00.748737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:00.748737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:00.748737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:00.748737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:00.748737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:00.748737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.748737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:00.748737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.748974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:00.748974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:00.748974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:00.748974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:00.748974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:00.748974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.748974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:00.748974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:00.748974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.748974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:00.748974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:00.748974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.748974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:00.748974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:00.748974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.748974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:00.748974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:00.748974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:00.748974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.749376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:00.755287 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:00.755792 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:00.755877 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:00.756021 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:00.756458 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:00.756699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:00.756857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.758788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:00.758788 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:00.758788 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:00.758788 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:00.758788 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:00.758788 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:00.758788 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.758788 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:00.759581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.759581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:00.759581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:00.759581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:00.759581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.759581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:00.759581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:00.759581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.759581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:00.759581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:00.759581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:00.759581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.760246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:00.760246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:00.760246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:00.760246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:00.760246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:00.760246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:00.760246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:00.760246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:00.760246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:00.760246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:00.760246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:00.760246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:00.760246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:00.760519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.760519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:00.760519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.760519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:00.760519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:00.760519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:00.760519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:00.760519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:00.760519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.760519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:00.760519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:00.760519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.760519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:00.760519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:00.760519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.760784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:00.760784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:00.760784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:00.760784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:00.760784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:00.760784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.760784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:00.760784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:00.760784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:00.760784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:00.760784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:00.760784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:00.760784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:00.760784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:00.760784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:00.760784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:00.760784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:00.760784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:00.760784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:00.761341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:00.761341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:00.761341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:00.761341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:00.761341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:00.761341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:03.648479 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:03.648871 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:03.648977 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:03.649974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:03.680593 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:03.680787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:03.681399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:03.681399 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:03.681399 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:03.681399 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:03.681399 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:03.681399 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:03.681399 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:03.681399 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:03.682668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:03.682668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:03.682668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:03.682668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:03.682668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:03.682668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:03.682668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:03.682668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:03.682668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:03.682668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:03.682668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:03.682668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:03.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:03.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:03.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:03.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:03.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:03.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:03.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:03.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:03.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:03.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:03.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:03.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:03.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:03.683460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:03.683460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:03.683460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:03.683460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:03.683460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:03.683460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:03.683460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:03.683460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:03.683460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:03.683460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:03.683460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:03.683460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:03.683460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:03.683460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:03.683460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:03.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:03.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:03.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:03.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:03.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:03.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:03.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:03.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:03.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:03.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:03.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:03.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:03.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:03.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:03.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:03.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:03.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:03.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:03.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:03.684625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:03.684625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:03.684625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:03.684625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:03.684625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:03.684625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:05.718238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:05.718698 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:05.718698 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:05.719681 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:05.719866 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:05.719984 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:05.720113 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:05.720240 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:05.720392 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:05.720515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:05.720628 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:05.720740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.721557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:05.721557 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:05.721557 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:05.721557 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:05.721557 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:05.721557 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:05.721557 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.721557 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:05.722687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.722687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:05.722687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:05.722687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:05.722687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.722687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:05.722687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:05.722687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.722687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:05.722687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:05.722687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:05.722687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.723118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:05.723118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:05.723118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:05.723118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:05.723118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:05.723118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:05.723118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:05.723118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:05.723118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:05.723118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:05.723118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:05.723118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:05.723118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:05.723588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.723588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:05.723588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.723588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:05.723588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:05.723588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:05.723588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:05.723588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:05.723588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.723588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:05.723588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:05.723588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.723588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:05.723588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:05.723588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.723924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:05.723924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:05.723924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.723924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:05.723924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:05.723924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.723924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:05.723924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:05.723924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:05.723924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.723924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:05.723924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:05.723924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:05.723924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:05.723924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:05.723924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:05.723924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:05.723924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:05.723924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:05.724368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:05.724368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:05.724368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:05.724368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:05.724368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.724368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:05.734673 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:05.735202 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:05.735347 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:05.735475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:28:05.736264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:05.736459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.736999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:05.736999 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:05.736999 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:05.736999 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:05.736999 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:05.736999 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:05.736999 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.736999 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:05.737637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.737637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:05.737637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:05.737637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:05.737637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.737637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:05.737637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:05.737637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:05.737637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.737637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:05.737637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:05.737637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:05.737637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:05.737637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:05.737637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:05.738005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:05.738005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:05.738005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:05.738005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:05.738005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:05.738005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:05.738005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:05.738005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.738005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:05.738247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.738247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:05.738247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:05.738247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:05.738247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.738247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:05.738247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:05.738247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:28:05.738247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:05.738247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:05.738247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.738247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:05.738612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:05.738612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:05.738612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.738612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:05.738612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:05.738612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:05.738612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:05.738612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:05.738612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:05.738612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:05.738612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:05.738612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:05.738612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:05.738612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:05.738904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:05.738904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:05.738904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.738904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:05.739018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.739018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:05.739018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:05.739018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:05.739018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:05.739018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:05.739018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.739018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:05.739018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:05.739018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.739018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:05.739018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:05.739018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.739018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:05.739018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:05.739018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.739562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:05.739562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:05.739562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:28:05.739562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:05.739562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:05.739562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.739562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:05.739562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:05.739562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:05.739562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.739562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:05.739562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:05.739562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:05.739562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:05.739562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:05.739562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:05.739562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:05.739562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:05.739940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:05.739940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:05.739940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:05.739940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:05.739940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:05.739940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.739940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:05.745967 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:05.746918 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:05.747150 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:05.747304 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:05.748793 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:05.749044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.749624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:05.749624 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:05.749624 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:05.749624 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:05.749624 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:05.749624 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:05.749624 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.749624 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:05.750240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.750240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:05.750240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:05.750240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:05.750240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.750240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:05.750240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:05.750240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:05.750240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.750240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:05.750240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:05.750240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:05.750240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:05.750240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:05.750240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:05.750562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:05.750562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:05.750562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:05.750562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:05.750562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:05.750562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:05.750562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:05.750562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.750562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:05.750856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.750856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:05.750856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:05.750856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:05.750856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.750856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:05.750856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:05.750856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:05.750856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:05.750856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.750856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:05.751075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:05.751075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:05.751075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.751075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:05.751075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:05.751075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:05.751075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:05.751075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:05.751075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:05.751075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:05.751075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:05.751075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:05.751075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:05.751075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:05.751453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:05.751453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:05.751453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.751453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:05.751909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.751909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:05.751909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:05.751909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:05.751909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:05.751909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:05.751909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.751909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:05.751909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:05.751909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.751909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:05.751909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:05.751909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.751909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:05.751909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:05.751909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.751909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:05.752321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:05.752321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:05.752321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:05.752321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.752321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:05.752321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:05.752321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:05.752321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.752321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:05.752321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:05.752321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:05.752321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:05.752321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:05.752321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:05.752321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:05.752321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:05.752321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:05.752321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:05.752321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:05.752631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:05.752631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:05.752631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.752631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:05.756310 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:05.756437 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:05.756512 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:05.756651 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:05.758469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:05.758739 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:05.758978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.759421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:05.759421 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:05.759421 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:05.759421 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:05.759421 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:05.759421 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:05.759421 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.759421 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:05.759967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.759967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:05.759967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:05.759967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:05.759967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.759967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:05.759967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:05.759967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.759967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:05.759967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:05.759967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:05.759967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.760369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:05.760369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:05.760369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:05.760369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:05.760369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:05.760369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:05.760369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:05.760369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:05.760369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:05.760369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:05.760369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:05.760369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:05.760369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:05.761736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.761736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:05.761736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.761736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:05.761736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:05.761736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:05.761736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:05.761736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:05.761736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.761736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:05.761736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:05.761736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.761736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:05.761736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:05.761736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.762093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:05.762093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:05.762093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:05.762093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:05.762093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:05.762093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.762093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:05.762093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:05.762093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:05.762093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:05.762093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:05.762093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:05.762093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:05.762093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:05.762093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:05.762093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:05.762093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:05.762093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:05.762093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:05.762536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:05.762536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:05.762536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:05.762536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:05.762536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:05.762536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:08.648862 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:08.649407 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:08.649548 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:08.650558 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:08.682283 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:08.682622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:08.683351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:08.683351 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:08.683351 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:08.683351 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:08.683351 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:08.683351 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:08.683351 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:08.683351 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:08.684078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:08.684078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:08.684078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:08.684078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:08.684078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:08.684078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:08.684078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:08.684078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:08.684078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:08.684078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:08.684078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:08.684078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:08.684360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:08.684360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:08.684360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:08.684360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:08.684360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:08.684360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:08.684360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:08.684360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:08.684360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:08.684360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:08.684360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:08.684360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:08.684360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:08.684613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:08.684613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:08.684843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:08.684843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:08.684843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:08.684843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:08.684843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:08.684843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:08.684843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:08.684843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:08.684843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:08.684843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:08.684843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:08.684843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:08.684843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:08.684843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:08.684843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:08.684843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:08.685322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:08.685322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:08.685322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:08.685322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:08.685322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:08.685322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:08.685322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:08.685322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:08.685322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:08.685322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:08.685322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:08.685322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:08.685322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:08.685322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:08.685322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:08.685322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:08.685322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:08.685322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:08.685322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:08.685710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:08.685710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:08.685710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:10.718758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:10.719378 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:10.719454 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:10.720364 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:10.720623 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:10.720856 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:10.721263 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:10.721544 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:10.722390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:10.723278 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:10.723526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:10.723748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-49414b01-83df-4428-b0f5-da55f126f6ca, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:10.723969 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:10.724196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.724827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:10.724827 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:10.724827 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:10.724827 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:10.724827 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:10.724827 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:10.724827 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.724827 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:10.725461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.725461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:10.725461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:10.725461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:10.725461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.725461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:10.725461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:10.725461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.725461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:10.725461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:10.725461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:10.725461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.725974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:10.725974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:10.725974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:10.725974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:10.725974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:10.725974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:10.725974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:10.725974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:10.725974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:10.725974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:10.725974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:10.725974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:10.725974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:10.726488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.726488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:10.726488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.726488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:10.726488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:10.726488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:10.726488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:10.726488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:10.726488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.726488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:10.726488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:10.726488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.726488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:10.726488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:10.726488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.727121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:10.727121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:10.727121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.727121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:10.727121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:10.727121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.727121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:10.727121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:10.727121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:10.727121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.727121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:10.727121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:10.727121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:10.727121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:10.727121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:10.727121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:10.727121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:10.727121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:10.727121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:10.727609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:10.727609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:10.727609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:10.727609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:10.727609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.727609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:10.735145 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:10.735631 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:10.736992 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:10.737126 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:28:10.737535 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:10.740769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.741224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:10.741224 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:10.741224 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:10.741224 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:10.741224 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:10.741224 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:10.741224 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.741224 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:10.742002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.742002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:10.742002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:10.742002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:10.742002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.742002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:10.742002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:10.742002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:10.742002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.742002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:10.742002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:10.742002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:10.742002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:10.742002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:10.742002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:10.742658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:10.742658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:10.742658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:10.742658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:10.742658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:10.742658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:10.742658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:10.742658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.742658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:10.742658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.742658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:10.742658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:10.742658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:10.742658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.743126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:10.743126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:10.743126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:28:10.743126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:10.743126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:10.743126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.743126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:10.743126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:10.743126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:10.743126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.743126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:10.743126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:10.743126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:10.743644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:10.743644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:10.743644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:10.743644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:10.743644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:10.743644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:10.743644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:10.743644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:10.743644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:10.743644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:10.743644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.743644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:10.743948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.743948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:10.743948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:10.743948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:10.743948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:10.743948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:10.743948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.743948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:10.743948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:10.743948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.743948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:10.743948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:10.743948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.743948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:10.743948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:10.743948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.744392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:10.744392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:10.744392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:28:10.744392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:10.744392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:10.744392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.744392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:10.744392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:10.744392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:10.744392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.744392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:10.744392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:10.744392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:10.744392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:10.744392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:10.744392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:10.744392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:10.744392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:10.744980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:10.744980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:10.744980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:10.744980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:10.744980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:10.744980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.744980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:10.746457 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:10.748310 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:10.748416 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:10.750197 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:10.750505 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:10.750709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.751124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:10.751124 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:10.751124 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:10.751124 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:10.751124 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:10.751124 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:10.751124 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.751124 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:10.751871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.751871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:10.751871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:10.751871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:10.751871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.751871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:10.751871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:10.751871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:10.751871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.751871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:10.751871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:10.751871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:10.751871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:10.751871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:10.751871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:10.752357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:10.752357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:10.752357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:10.752357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:10.752357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:10.752357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:10.752357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:10.752357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.752357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:10.752873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.752873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:10.752873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:10.752873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:10.752873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.752873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:10.752873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:10.752873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:10.752873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:10.752873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.752873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:10.753349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:10.753349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:10.753349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.753349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:10.753349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:10.753349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:10.753349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:10.753349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:10.753349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:10.753349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:10.753349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:10.753349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:10.753349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:10.753349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:10.753772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:10.753772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:10.753772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.753772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:10.754054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.754054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:10.754054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:10.754054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:10.754054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:10.754054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:10.754054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.754054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:10.754054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:10.754054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.754054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:10.754054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:10.754054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.754054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:10.754054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:10.754054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.754054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:10.754395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:10.754395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:10.754395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:10.754395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.754395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:10.754395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:10.754395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:10.754395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.754395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:10.754395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:10.754395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:10.754395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:10.754395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:10.754395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:10.754395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:10.754395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:10.754395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:10.754395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:10.754395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:10.754720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:10.754720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:10.754720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.754720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:10.756424 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:10.756653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:10.756769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:10.756941 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:10.757297 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:10.757532 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:10.757763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.758132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:10.758132 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:10.758132 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:10.758132 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:10.758132 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:10.758132 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:10.758132 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.758132 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:10.758820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.758820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:10.758820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:10.758820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:10.758820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.758820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:10.758820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:10.758820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.758820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:10.758820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:10.758820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:10.758820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.759215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:10.759215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:10.759215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:10.759215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:10.759215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:10.759215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:10.759215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:10.759215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:10.759215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:10.759215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:10.759215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:10.759215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:10.759215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:10.759641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.759641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:10.759849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.759849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:10.759849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:10.759849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:10.759849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:10.759849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:10.759849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.759849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:10.759849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:10.759849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.759849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:10.759849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:10.759849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.759849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:10.759849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:10.759849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:10.760169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:10.760169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:10.760169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.760169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:10.760169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:10.760169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:10.760169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:10.760169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:10.760169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:10.760169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:10.760169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:10.760169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:10.760169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:10.760169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:10.760169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:10.760169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:10.760169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:10.760169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:10.760169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:10.760506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:10.760506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:10.760506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:13.649465 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:13.650112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:13.650112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:13.650639 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:13.712829 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:13.712829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:13.712829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:13.712829 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:13.712829 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:13.712829 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:13.712829 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:13.712829 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:13.712829 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:13.712829 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:13.712829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.061 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:13.712829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:13.712829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:13.713303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:13.713303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:13.713303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:13.713303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:13.713303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:13.713303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:13.713303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:13.713303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:13.713303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:13.713303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:13.713303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:13.713303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:13.713303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:13.713303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:13.713601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:13.713601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:13.713601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:13.713601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:13.713601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:13.713601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:13.713601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:13.713601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:13.713601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:13.713601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:13.713601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:13.713601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:13.714307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:13.714307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:13.714307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:13.714307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:13.714307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:13.714307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:13.714307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:13.714307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:13.714307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:13.714307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:13.714307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:13.714307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:13.714307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:13.714307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:13.714307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:13.714307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:13.714307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:13.714307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:13.714307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:13.715143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:13.715143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:13.715143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:13.715143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:13.715143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:13.715143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:13.715143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:13.715143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:13.715143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:13.715143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:13.715143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:13.715143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:13.715143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:13.715143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:13.715143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:13.715143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:13.715143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:15.721449 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:15.722843 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:15.722933 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:15.724411 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:15.724547 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:15.724672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:15.724774 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:15.724881 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:15.725009 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:15.725122 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:15.725216 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:15.725340 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-49414b01-83df-4428-b0f5-da55f126f6ca, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:15.725435 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:15.725527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.725963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:15.725963 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:15.725963 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:15.725963 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:15.725963 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:15.725963 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:15.725963 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.725963 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:15.726635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.726635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:15.726635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:15.726635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:15.726635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.726635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:15.726635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:15.726635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.726635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:15.726635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:15.726635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:15.726635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.726903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:15.726903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:15.726903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:15.726903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:15.726903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:15.726903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:15.726903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:15.726903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:15.726903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:15.726903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:15.726903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:15.726903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:15.726903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:15.727170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.727170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:15.728791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.728791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:15.728791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:15.728791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:15.728791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:15.728791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:15.728791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.728791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:15.728791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:15.728791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.728791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:15.728791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:15.728791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.728791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:15.728791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:15.728791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.729247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:15.729247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:15.729247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.729247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:15.729247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:15.729247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:15.729247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.729247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:15.729247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:15.729247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:15.729247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:15.729247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:15.729247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:15.729247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:15.729247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:15.729247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:15.729247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:15.729247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:15.729247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:15.729861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:15.729861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.729861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:15.736158 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:15.736995 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:15.737105 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:15.737199 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:28:15.737553 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:15.737716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.738378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:15.738378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:15.738378 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:15.738378 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:15.738378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:15.738378 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:15.738378 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.738378 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:15.738867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.738867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:15.738867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:15.738867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:15.738867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.738867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:15.738867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:15.738867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:15.738867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.738867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:15.738867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:15.738867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:15.738867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:15.738867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:15.738867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:15.739140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:15.739140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:15.739140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:15.739140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:15.739140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:15.739140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:15.739140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:15.739140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.739140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:15.739521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.739521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:15.739521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:15.739521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:15.739521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.739521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:15.739521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:15.739521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:28:15.739521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:15.739521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:15.739521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.739521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:15.739760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:15.739760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:15.739760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.739760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:15.739760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:15.739760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:15.739760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:15.739760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:15.739760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:15.739760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:15.739760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:15.739760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:15.739760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:15.739760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:15.739980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:15.739980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:15.739980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.739980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:15.740926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.740926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:15.740926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:15.740926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:15.740926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:15.740926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:15.740926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.740926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:15.740926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:15.740926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.740926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:15.740926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:15.740926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.740926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:15.740926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:15.740926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.741270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:15.741270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:15.741270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:28:15.741270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:15.741270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:15.741270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.741270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:15.741270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:15.741270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:15.741270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.741270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:15.741270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:15.741270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:15.741270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:15.741270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:15.741270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:15.741270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:15.741270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:15.741667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:15.741667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:15.741667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:15.741667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:15.741667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:15.741667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.741667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:15.747525 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:15.748065 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:15.748224 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:15.748393 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:15.748733 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:15.748981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.749488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:15.749488 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:15.749488 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:15.749488 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:15.749488 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:15.749488 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:15.749488 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.749488 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:15.750246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.750246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:15.750246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:15.750246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:15.750246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.750246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:15.750246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:15.750246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:15.750246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.750246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:15.750246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:15.750246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:15.750246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:15.750246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:15.750246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:15.750553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:15.750553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:15.750553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:15.750553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:15.750553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:15.750553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:15.750553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:15.750553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.750553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:15.751191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.751191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:15.751191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:15.751191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:15.751191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.751191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:15.751191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:15.751191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:15.751191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:15.751191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.751191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:15.751441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:15.751441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:15.751441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.751441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:15.751441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:15.751441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:15.751441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:15.751441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:15.751441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:15.751441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:15.751441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:15.751441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:15.751441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:15.751441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:15.751848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:15.751848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:15.751848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.751848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:15.752410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.752410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:15.752410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:15.752410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:15.752410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:15.752410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:15.752410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.752410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:15.752410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:15.752410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.752410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:15.752410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:15.752410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.752410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:15.752410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:15.752410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.752410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:15.753164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:15.753164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:15.753164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:15.753164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.753164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:15.753164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:15.753164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:15.753164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.753164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:15.753164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:15.753164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:15.753164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:15.753164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:15.753164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:15.753164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:15.753164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:15.753164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:15.753164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:15.753164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:15.753530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:15.753530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:15.753530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.753530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:15.757829 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:15.757829 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:15.758011 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:15.759200 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:15.759200 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:15.759200 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:15.759200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.759200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:15.759200 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:15.759200 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:15.759200 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:15.759200 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:15.759200 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:15.759200 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.759200 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:15.759848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.759848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:15.759848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:15.759848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:15.759848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.759848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:15.759848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:15.759848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.759848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:15.759848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:15.759848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:15.759848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.760160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:15.760160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:15.760160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:15.760160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:15.760160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:15.760160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:15.760160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:15.760160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:15.760160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:15.760160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:15.760160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:15.760160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:15.760160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:15.760386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.760386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:15.760872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.760872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:15.760872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:15.760872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:15.760872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:15.760872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:15.760872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.760872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:15.760872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:15.760872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.760872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:15.760872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:15.760872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.760872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:15.760872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:15.760872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:15.761522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:15.761522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:15.761522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.761522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:15.761522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:15.761522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:15.761522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:15.761522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:15.761522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:15.761522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:15.761522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:15.761522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:15.761522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:15.761522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:15.761522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:15.761522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:15.761522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:15.761522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:15.761522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:15.761893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:15.761893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:15.761893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:18.649901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:18.653752 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:18.654109 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:18.655088 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:18.697001 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:18.697428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:18.698105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:18.698105 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:18.698105 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:18.698105 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:18.698105 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:18.698105 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:18.698105 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:18.698105 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:18.698824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:18.698824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:18.698824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:18.698824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:18.698824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:18.698824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:18.698824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:18.698824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:18.698824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:18.698824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:18.698824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:18.698824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:18.699117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:18.699117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:18.699117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:18.699117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:18.699117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:18.699117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:18.699117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:18.699117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:18.699117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:18.699117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:18.699117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:18.699117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:18.699117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:18.699394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:18.699394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:18.700659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:18.700659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:18.700659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:18.700659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:18.700659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:18.700659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:18.700659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:18.700659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:18.700659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:18.700659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:18.700659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:18.700659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:18.700659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:18.700659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:18.700659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:18.700659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:18.701282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:18.701282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:18.701282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:18.701282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:18.701282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:18.701282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:18.701282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:18.701282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:18.701282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:18.701282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:18.701282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:18.701282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:18.701282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:18.701282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:18.701282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:18.701282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:18.701282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:18.701282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:18.701282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:18.701742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:18.701742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:18.701742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:20.722956 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:20.723880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:20.723880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:20.725830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:20.725830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:20.725830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:20.725830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:20.725830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:20.726064 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:20.727000 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:20.727245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:20.727686 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-49414b01-83df-4428-b0f5-da55f126f6ca, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:20.727877 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:20.728096 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-56b741fb-a48b-491e-9a3c-cdef706732d6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:20.728323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.728994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:20.728994 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:20.728994 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:20.728994 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:20.728994 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:20.728994 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:20.728994 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.728994 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:20.730161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.730161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:20.730161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:20.730161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:20.730161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.730161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:20.730161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:20.730161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.730161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:20.730161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:20.730161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:20.730161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.730888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:20.730888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:20.730888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:20.730888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:20.730888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:20.730888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:20.730888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:20.730888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:20.730888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:20.730888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:20.730888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:20.730888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:20.730888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:20.731196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.731196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:20.731196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.731196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:20.731196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:20.731196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:20.731196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:20.731196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:20.731196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.731196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:20.731196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:20.731196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.731196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:20.731196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:20.731196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.731497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:20.731497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:20.731497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.731497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:20.731497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:20.731497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.731497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:20.731497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:20.731497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:20.731497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.731497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:20.731497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:20.731497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:20.731497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:20.731497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:20.731497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:20.731497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:20.731497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:20.731497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:20.731824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:20.731824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:20.731824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:20.731824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:20.731824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.731824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:20.737153 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:20.738134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:20.738303 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:20.738446 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:28:20.739553 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:20.739553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.739674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:20.739674 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:20.739674 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:20.739674 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:20.739674 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:20.739674 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:20.739674 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.739674 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:20.740683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.740683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:20.740683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:20.740683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:20.740683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.740683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:20.740683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:20.740683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:20.740683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.740683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:20.740683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:20.740683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:20.740683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:20.740683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:20.740683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:20.741036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:20.741036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:20.741036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:20.741036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:20.741036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:20.741036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:20.741036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:20.741036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.741036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:20.741624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.741624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:20.741624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:20.741624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:20.741624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.741624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:20.741624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:20.741624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:28:20.741624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:20.741624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:20.741624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.741624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:20.741926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:20.741926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:20.741926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.741926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:20.741926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:20.741926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:20.741926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:20.741926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:20.741926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:20.741926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:20.741926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:20.741926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:20.741926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:20.741926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:20.742229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:20.742229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:20.742229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.742229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:20.742765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.742765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:20.742765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:20.742765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:20.742765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:20.742765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:20.742765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.742765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:20.742765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:20.742765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.742765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:20.742765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:20.742765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.742765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:20.742765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:20.742765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.743122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:20.743122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:20.743122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:28:20.743122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:20.743122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:20.743122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.743122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:20.743122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:20.743122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:20.743122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.743122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:20.743122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:20.743122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:20.743122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:20.743122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:20.743122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:20.743122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:20.743122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:20.743549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:20.743549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:20.743549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:20.743549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:20.743549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:20.743549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.743549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:20.748124 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:20.749707 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:20.749856 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:20.750001 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:20.751039 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:20.751039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.751164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:20.751164 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:20.751164 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:20.751164 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:20.751164 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:20.751164 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:20.751164 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.751164 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:20.752010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.752010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:20.752010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:20.752010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:20.752010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.752010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:20.752010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:20.752010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:20.752010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.752010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:20.752010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:20.752010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:20.752010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:20.752010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:20.752010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:20.752353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:20.752353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:20.752353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:20.752353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:20.752353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:20.752353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:20.752353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:20.752353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.752353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:20.752958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.752958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:20.752958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:20.752958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:20.752958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.752958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:20.752958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:20.752958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:20.752958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:20.752958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.752958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:20.753207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:20.753207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:20.753207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.753207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:20.753207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:20.753207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:20.753207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:20.753207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:20.753207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:20.753207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:20.753207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:20.753207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:20.753207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:20.753207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:20.753499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:20.753499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:20.753499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.753499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:20.754089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.754089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:20.754089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:20.754089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:20.754089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:20.754089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:20.754089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.754089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:20.754089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:20.754089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.754089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:20.754089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:20.754089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.754089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:20.754089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:20.754089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.754089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:20.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:20.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:20.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:20.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:20.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:20.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:20.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:20.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:20.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:20.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:20.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:20.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:20.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:20.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:20.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:20.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:20.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:20.755004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:20.755004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:20.755004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.755004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:20.758342 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:20.758726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:20.758861 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:20.759092 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:20.759561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:20.759956 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:20.760434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.763539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:20.763539 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:20.763539 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:20.763539 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:20.763539 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:20.763539 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:20.763539 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.763539 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:20.764685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.764685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:20.764685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:20.764685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:20.764685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.764685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:20.764685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:20.764685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.764685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:20.764685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:20.764685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:20.764685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.764958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:20.764958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:20.764958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:20.764958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:20.764958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:20.764958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:20.764958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:20.764958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:20.764958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:20.764958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:20.764958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:20.764958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:20.764958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:20.765203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.765203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:20.765889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.765889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:20.765889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:20.765889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:20.765889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:20.765889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:20.765889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.765889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:20.765889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:20.765889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.765889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:20.765889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:20.765889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.765889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:20.765889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:20.765889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:20.766287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:20.766287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:20.766287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.766287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:20.766287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:20.766287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:20.766287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:20.766287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:20.766287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:20.766287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:20.766287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:20.766287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:20.766287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:20.766287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:20.766287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:20.766287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:20.766287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:20.766287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:20.766287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:20.766694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:20.766694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:20.766694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:23.653893 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:23.654455 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:23.654455 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:23.654871 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:23.693620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:23.694128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:23.694419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:23.694419 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:23.694419 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:23.694419 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:23.694419 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:23.694419 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:23.694419 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:23.694419 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:23.695182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:23.695182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:23.695182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:23.695182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:23.695182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:23.695182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:23.695182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:23.695182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:23.695182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:23.695182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:23.695182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:23.695182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:23.695601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:23.695601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:23.695601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:23.695601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:23.695601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:23.695601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:23.695601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:23.695601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:23.695601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:23.695601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:23.695601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:23.695601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:23.695601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:23.696100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:23.696100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:23.696459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:23.696459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:23.696459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:23.696459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:23.696459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:23.696459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:23.696459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:23.696459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:23.696459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:23.696459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:23.696459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:23.696459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:23.696459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:23.696459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:23.696459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:23.696459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:23.696997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:23.696997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:23.696997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:23.696997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:23.696997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:23.696997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:23.696997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:23.696997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:23.696997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:23.696997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:23.696997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:23.696997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:23.696997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:23.696997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:23.696997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:23.696997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:23.696997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:23.696997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:23.696997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:23.697591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:23.697591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:23.697591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:25.723430 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:25.723914 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:25.724092 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:25.725241 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:25.725570 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:25.725638 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:25.725734 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:25.725836 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:25.725950 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:25.726056 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:25.726167 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-49414b01-83df-4428-b0f5-da55f126f6ca, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:25.726344 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:25.726551 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-56b741fb-a48b-491e-9a3c-cdef706732d6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:25.726663 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:25.726769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.729139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:25.729139 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:25.729139 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:25.729139 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:25.729139 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:25.729139 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:25.729139 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.729139 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:25.729922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.729922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:25.729922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:25.729922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:25.729922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.729922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:25.729922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:25.729922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.729922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:25.729922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:25.729922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:25.729922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.730312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:25.730312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:25.730312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:25.730312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:25.730312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:25.730312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:25.730312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:25.730312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:25.730312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:25.730312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:25.730312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:25.730312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:25.730312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:25.730724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.730724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:25.731169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.731169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:25.731169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:25.731169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:25.731169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:25.731169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:25.731169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.731169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:25.731169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:25.731169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.731169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:25.731169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:25.731169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.731169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:25.731169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:25.731169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.731612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:25.731612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:25.731612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.731612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:25.731612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:25.731612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:25.731612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.731612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:25.731612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:25.731612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:25.731612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:25.731612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:25.731612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:25.731612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:25.731612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:25.731612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:25.731612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:25.731612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:25.731612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:25.732164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:25.732164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.732164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:25.740487 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:25.740852 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:25.741005 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:25.741212 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:28:25.743611 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:25.743807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.744299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:25.744299 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:25.744299 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:25.744299 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:25.744299 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:25.744299 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:25.744299 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.744299 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:25.744960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.744960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:25.744960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:25.744960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:25.744960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.744960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:25.744960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:25.744960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:25.744960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.744960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:25.744960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:25.744960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:25.744960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:25.744960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:25.744960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:25.745465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:25.745465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:25.745465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:25.745465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:25.745465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:25.745465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:25.745465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:25.745465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.745465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:25.745741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.745741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:25.745741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:25.745741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:25.745741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.745741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:25.745741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:25.745741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:28:25.745741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:25.745741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:25.745741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.745741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:25.746143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:25.746143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:25.746143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.746143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:25.746143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:25.746143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:25.746143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:25.746143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:25.746143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:25.746143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:25.746143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:25.746143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:25.746143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:25.746143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:25.746532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:25.746532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:25.746532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.746532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:25.746532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.746532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:25.746532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:25.746532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:25.746532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:25.746532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:25.746532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.746532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:25.746532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:25.746532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.746951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:25.746951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:25.746951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.746951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:25.746951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:25.746951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.746951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:25.746951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:25.746951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:28:25.746951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:25.746951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:25.746951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.746951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:25.746951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:25.746951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:25.746951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.746951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:25.746951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:25.746951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:25.747480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:25.747480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:25.747480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:25.747480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:25.747480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:25.747480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:25.747480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:25.747480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:25.747480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:25.747480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:25.747480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.747480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:25.749691 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:25.749953 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:25.750073 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:25.750197 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:25.750577 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:25.750791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.751248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:25.751248 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:25.751248 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:25.751248 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:25.751248 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:25.751248 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:25.751248 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.751248 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:25.751957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.751957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:25.751957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:25.751957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:25.751957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.751957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:25.751957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:25.751957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:25.751957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.751957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:25.751957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:25.751957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:25.751957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:25.751957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:25.751957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:25.752428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:25.752428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:25.752428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:25.752428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:25.752428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:25.752428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:25.752428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:25.752428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.752428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:25.752708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.752708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:25.752708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:25.752708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:25.752708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.752708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:25.752708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:25.752708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:25.752708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:25.752708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.752708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:25.752985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:25.752985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:25.752985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.752985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:25.752985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:25.752985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:25.752985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:25.752985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:25.752985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:25.752985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:25.752985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:25.752985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:25.752985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:25.752985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:25.753367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:25.753367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:25.753367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.753367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:25.753492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.753492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:25.753492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:25.753492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:25.753492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:25.753492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:25.753492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.753492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:25.753492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:25.753492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.753492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:25.753492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:25.753492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.753492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:25.753492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:25.753492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.753492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:25.753947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:25.753947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:25.753947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:25.753947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.753947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:25.753947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:25.753947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:25.753947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.753947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:25.753947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:25.753947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:25.753947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:25.753947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:25.753947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:25.753947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:25.753947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:25.753947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:25.753947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:25.753947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:25.754481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:25.754481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:25.754481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.754481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:25.758691 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:25.758929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:25.759056 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:25.759201 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:25.759558 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:25.759774 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:25.760065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.760527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:25.760527 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:25.760527 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:25.760527 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:25.760527 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:25.760527 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:25.760527 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.760527 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:25.761218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.761218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:25.761218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:25.761218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:25.761218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.761218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:25.761218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:25.761218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.761218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:25.761218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:25.761218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:25.761218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.761557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:25.761557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:25.761557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:25.761557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:25.761557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:25.761557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:25.761557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:25.761557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:25.761557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:25.761557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:25.761557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:25.761557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:25.761557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:25.761960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.761960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:25.762138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.762138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:25.762138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:25.762138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:25.762138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:25.762138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:25.762138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.762138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:25.762138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:25.762138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.762138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:25.762138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:25.762138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.762138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:25.762138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:25.762138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:25.762517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:25.762517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:25.762517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.762517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:25.762517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:25.762517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:25.762517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:25.762517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:25.762517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:25.762517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:25.762517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:25.762517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:25.762517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:25.762517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:25.762517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:25.762517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:25.762517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:25.762517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:25.762517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:25.763068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:25.763068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:25.763068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:28.654496 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:28.654951 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:28.655060 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:28.655705 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:28.718619 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:28.718824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:28.719528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:28.719528 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:28.719528 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:28.719528 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:28.719528 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:28.719528 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:28.719528 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:28.719528 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:28.720563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.064 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:28.720563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:28.720563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:28.720563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:28.720563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:28.720563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:28.720563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:28.720563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:28.720563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:28.720563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:28.720563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:28.720563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:28.720909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:28.720909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:28.720909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:28.720909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:28.720909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:28.720909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:28.720909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:28.720909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:28.720909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:28.720909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:28.720909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:28.720909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:28.720909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:28.721298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:28.721298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:28.721298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:28.721298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:28.721298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:28.721298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:28.721298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:28.721298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:28.721298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:28.721298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:28.721298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:28.721298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:28.721298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:28.721298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:28.721298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:28.721741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:28.721741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:28.721741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:28.721741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:28.721741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:28.721741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:28.721741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:28.721741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:28.721741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:28.721741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:28.721741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:28.721741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:28.721741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:28.721741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:28.721741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:28.721741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:28.721741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:28.721741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:28.721741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:28.722387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:28.722387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:28.722387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:28.722387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:28.722387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:28.722387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:30.725538 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:30.726110 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:30.726110 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:30.730739 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:30.730929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:30.732375 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:30.733164 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:30.733627 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:30.733627 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:30.733627 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-49414b01-83df-4428-b0f5-da55f126f6ca, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:30.733727 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:30.733766 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-56b741fb-a48b-491e-9a3c-cdef706732d6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:30.733869 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:30.733955 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:30.734099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.734555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:30.734555 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:30.734555 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:30.734555 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:30.734555 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:30.734555 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:30.734555 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.734555 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:30.735285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.735285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:30.735285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:30.735285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:30.735285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.735285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:30.735285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:30.735285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.735285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:30.735285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:30.735285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:30.735285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.736018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:30.736018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:30.736018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:30.736018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:30.736018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:30.736018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:30.736018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:30.736018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:30.736018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:30.736018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:30.736018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:30.736018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:30.736018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:30.736365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.736365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:30.736365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.736365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:30.736365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:30.736365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:30.736365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:30.736365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:30.736365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.736365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:30.736365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:30.736365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.736365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:30.736365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:30.736365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.736693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:30.736693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:30.736693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.736693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:30.736693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:30.736693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.736693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:30.736693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:30.736693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:30.736693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.736693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:30.736693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:30.736693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:30.736693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:30.736693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:30.736693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:30.736693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:30.736693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:30.736693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:30.737103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:30.737103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:30.737103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:30.737103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:30.737103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.737103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:30.740940 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:30.741610 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:30.741758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:30.741907 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:28:30.743562 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:30.743562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.745489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:30.745489 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:30.745489 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:30.745489 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:30.745489 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:30.745489 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:30.745489 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.745489 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:30.746351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.746351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:30.746351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:30.746351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:30.746351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.746351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:30.746351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:30.746351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:30.746351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.746351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:30.746351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:30.746351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:30.746351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:30.746351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:30.746351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:30.746875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:30.746875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:30.746875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:30.746875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:30.746875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:30.746875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:30.746875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:30.746875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.746875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:30.747420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.747420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:30.747420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:30.747420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:30.747420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.747420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:30.747420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:30.747420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:28:30.747420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:30.747420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:30.747420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.747420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:30.748439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:30.748439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:30.748439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.748439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:30.748439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:30.748439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:30.748439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:30.748439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:30.748439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:30.748439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:30.748439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:30.748439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:30.748439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:30.748439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:30.751501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:30.751501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:30.751501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.751501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:30.753514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.753514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:30.753514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:30.753514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:30.753514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:30.753514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:30.753514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.753514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:30.753514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:30.753514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.753514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:30.753514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:30.753514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.753514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:30.753514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:30.753514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.753932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:30.753932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:30.753932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:28:30.753932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:30.753932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:30.753932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.753932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:30.753932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:30.753932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:30.753932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.753932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:30.753932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:30.753932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:30.753932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:30.753932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:30.753932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:30.753932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:30.753932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:30.754322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:30.754322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:30.754322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:30.754322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:30.754322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:30.754322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.754322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:30.754322 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:30.754322 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:30.754322 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:30.755175 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:30.755175 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:30.755175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.755175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:30.755175 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:30.755175 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:30.755175 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:30.755175 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:30.755175 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:30.755175 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.755175 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:30.755650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.755650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:30.755650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:30.755650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:30.755650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.755650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:30.755650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:30.755650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:30.755650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.755650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:30.755650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:30.755650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:30.755650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:30.755650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:30.755650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:30.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:30.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:30.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:30.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:30.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:30.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:30.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:30.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:30.756174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.756174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:30.756174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:30.756174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:30.756174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.756174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:30.756174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:30.756174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:30.756174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:30.756174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.756174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:30.756441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:30.756441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:30.756441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.756441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:30.756441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:30.756441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:30.756441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:30.756441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:30.756441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:30.756441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:30.756441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:30.756441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:30.756441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:30.756441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:30.756706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:30.756706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:30.756706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.756706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:30.757120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.757120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:30.757120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:30.757120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:30.757120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:30.757120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:30.757120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.757120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:30.757120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:30.757120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.757120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:30.757120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:30.757120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.757120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:30.757120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:30.757120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.757120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:30.757933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:30.757933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:30.757933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:30.757933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.757933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:30.757933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:30.757933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:30.757933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.757933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:30.757933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:30.757933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:30.757933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:30.757933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:30.757933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:30.757933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:30.757933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:30.757933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:30.757933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:30.757933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:30.759391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:30.759391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:30.759391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.759391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:30.759391 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:30.762526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:30.762686 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:30.762781 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:30.763617 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:30.764229 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:30.764717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.765509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:30.765509 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:30.765509 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:30.765509 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:30.765509 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:30.765509 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:30.765509 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.765509 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:30.766350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.766350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:30.766350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:30.766350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:30.766350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.766350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:30.766350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:30.766350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.766350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:30.766350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:30.766350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:30.766350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.766811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:30.766811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:30.766811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:30.766811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:30.766811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:30.766811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:30.766811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:30.766811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:30.766811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:30.766811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:30.766811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:30.766811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:30.766811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:30.767294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.767294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:30.767450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.767450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:30.767450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:30.767450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:30.767450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:30.767450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:30.767450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.767450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:30.767450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:30.767450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.767450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:30.767450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:30.767450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.767450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:30.767450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:30.767450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:30.767969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:30.767969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:30.767969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.767969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:30.767969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:30.767969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:30.767969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:30.767969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:30.767969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:30.767969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:30.767969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:30.767969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:30.767969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:30.767969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:30.767969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:30.767969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:30.767969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:30.767969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:30.767969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:30.768606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:30.768606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:30.768606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:33.654956 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:33.656651 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:33.656762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:33.663718 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:33.717973 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:33.718685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:33.718886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:33.718886 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:33.718886 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:33.718886 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:33.718886 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:33.718886 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:33.718886 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:33.718886 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:33.720014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.062 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:33.720014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:33.720014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:33.720014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:33.720014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:33.720014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:33.720014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:33.720014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:33.720014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:33.720014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:33.720014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:33.720014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:33.720742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:33.720742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:33.720742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:33.720742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:33.720742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:33.720742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:33.720742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:33.720742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:33.720742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:33.720742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:33.720742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:33.720742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:33.720742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:33.721150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:33.721150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:33.721233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:33.721233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:33.721233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:33.721233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:33.721233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:33.721233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:33.721233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:33.721233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:33.721233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:33.721233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:33.721233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:33.721233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:33.721233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:33.721233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:33.721233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:33.721233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:33.721667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:33.721667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:33.721667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:33.721667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:33.721667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:33.721667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:33.721667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:33.721667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:33.721667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:33.721667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:33.721667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:33.721667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:33.721667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:33.721667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:33.721667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:33.721667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:33.721667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:33.721667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:33.721667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:33.722206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:33.722206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:33.722206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:35.725961 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:35.726384 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:35.726462 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:35.728507 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:35.728699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:35.728865 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:35.729017 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:35.729166 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:35.730170 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-49414b01-83df-4428-b0f5-da55f126f6ca, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:35.730631 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:35.731061 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-56b741fb-a48b-491e-9a3c-cdef706732d6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:35.731061 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:35.731124 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:35.731267 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-408abcf3-eb63-4f99-b28f-1736dad8caf3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:35.731417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.732141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:35.732141 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:35.732141 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:35.732141 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:35.732141 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:35.732141 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:35.732141 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.732141 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:35.733127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.733127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:35.733127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:35.733127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:35.733127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.733127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:35.733127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:35.733127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.733127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:35.733127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:35.733127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:35.733127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.733855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:35.733855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:35.733855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:35.733855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:35.733855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:35.733855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:35.733855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:35.733855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:35.733855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:35.733855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:35.733855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:35.733855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:35.733855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:35.734107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.734107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:35.736645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.736645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:35.736645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:35.736645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:35.736645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:35.736645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:35.736645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.736645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:35.736645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:35.736645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.736645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:35.736645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:35.736645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.736645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:35.736645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:35.736645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.737647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:35.737647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:35.737647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.737647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:35.737647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:35.737647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:35.737647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.737647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:35.737647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:35.737647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:35.737647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:35.737647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:35.737647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:35.737647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:35.737647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:35.737647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:35.737647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:35.737647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:35.737647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:35.738191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:35.738191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.738191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:35.741675 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:35.742497 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:35.742497 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:35.742497 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:28:35.742966 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:35.743579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.744169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:35.744169 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:35.744169 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:35.744169 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:35.744169 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:35.744169 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:35.744169 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.744169 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:35.746105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.746105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:35.746105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:35.746105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:35.746105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.746105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:35.746105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:35.746105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:35.746105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.746105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:35.746105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:35.746105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:35.746105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:35.746105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:35.746105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:35.746596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:35.746596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:35.746596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:35.746596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:35.746596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:35.746596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:35.746596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:35.746596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.746596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:35.748316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.748316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:35.748316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:35.748316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:35.748316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.748316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:35.748316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:35.748316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:28:35.748316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:35.748316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:35.748316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.748316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:35.748811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:35.748811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:35.748811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.748811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:35.748811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:35.748811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:35.748811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:35.748811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:35.748811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:35.748811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:35.748811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:35.748811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:35.748811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:35.748811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:35.749268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:35.749268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:35.749268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.749268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:35.750680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.750680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:35.750680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:35.750680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:35.750680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:35.750680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:35.750680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.750680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:35.750680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:35.750680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.750680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:35.750680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:35.750680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.750680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:35.750680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:35.750680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.751269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:35.751269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:35.751269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:28:35.751269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:35.751269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:35.751269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.751269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:35.751269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:35.751269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:35.751269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.751269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:35.751269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:35.751269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:35.751269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:35.751269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:35.751269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:35.751269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:35.751269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:35.751807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:35.751807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:35.751807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:35.751807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:35.751807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:35.751807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.751807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:35.754547 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:35.755187 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:35.755187 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:35.755187 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:35.756289 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:35.756289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.756865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:35.756865 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:35.756865 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:35.756865 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:35.756865 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:35.756865 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:35.756865 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.756865 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:35.758331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.758331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:35.758331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:35.758331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:35.758331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.758331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:35.758331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:35.758331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:35.758331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.758331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:35.758331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:35.758331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:35.758331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:35.758331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:35.758331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:35.759344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:35.759344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:35.759344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:35.759344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:35.759344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:35.759344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:35.759344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:35.759344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.759344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:35.760505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.760505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:35.760505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:35.760505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:35.760505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.760505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:35.760505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:35.760505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:35.760505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:35.760505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.760505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:35.760758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:35.760758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:35.760758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.760758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:35.760758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:35.760758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:35.760758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:35.760758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:35.760758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:35.760758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:35.760758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:35.760758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:35.760758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:35.760758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:35.761444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:35.761444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:35.761444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.761444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:35.763169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.763169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:35.763169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:35.763169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:35.763169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:35.763169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:35.763169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.763169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:35.763169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:35.763169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.763169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:35.763169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:35.763169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.763169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:35.763169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:35.763169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.763169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:35.764547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:35.764547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:35.764547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:35.764547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.764547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:35.764547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:35.764547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:35.764547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.764547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:35.764547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:35.764547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:35.764547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:35.764547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:35.764547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:35.764547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:35.764547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:35.764547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:35.764547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:35.764547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:35.764901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:35.764901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:35.764901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.764901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:35.764901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:35.764901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:35.764901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:35.764901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:35.765104 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:35.765104 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:35.765104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.765428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:35.765428 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:35.765428 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:35.765428 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:35.765428 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:35.765428 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:35.765428 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.765428 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:35.766887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.766887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:35.766887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:35.766887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:35.766887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.766887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:35.766887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:35.766887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.766887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:35.766887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:35.766887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:35.766887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.767297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:35.767297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:35.767297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:35.767297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:35.767297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:35.767297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:35.767297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:35.767297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:35.767297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:35.767297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:35.767297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:35.767297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:35.767297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:35.767588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.767588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:35.768973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.768973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:35.768973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:35.768973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:35.768973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:35.768973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:35.768973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.768973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:35.768973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:35.768973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.768973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:35.768973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:35.768973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.768973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:35.768973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:35.768973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:35.769414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:35.769414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:35.769414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.769414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:35.769414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:35.769414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:35.769414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:35.769414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:35.769414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:35.769414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:35.769414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:35.769414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:35.769414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:35.769414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:35.769414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:35.769414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:35.769414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:35.769414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:35.769414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:35.769831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:35.769831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:35.769831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:38.656662 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:38.657484 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:38.657555 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:38.658147 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:38.715758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:38.715949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:38.716403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:38.716403 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:38.716403 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:38.716403 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:38.716403 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:38.716403 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:38.716403 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:38.716403 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:38.717870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.059 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:38.717870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:38.717870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:38.717870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:38.717870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:38.717870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:38.717870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:38.717870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:38.717870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:38.717870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:38.717870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:38.717870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:38.718435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:38.718435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:38.718435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:38.718435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:38.718435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:38.718435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:38.718435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:38.718435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:38.718435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:38.718435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:38.718435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:38.718435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:38.718435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:38.718901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:38.718901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:38.718901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:38.718901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:38.718901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:38.718901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:38.718901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:38.718901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:38.718901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:38.718901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:38.718901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:38.718901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:38.718901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:38.718901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:38.718901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:38.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:38.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:38.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:38.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:38.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:38.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:38.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:38.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:38.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:38.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:38.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:38.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:38.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:38.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:38.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:38.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:38.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:38.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:38.719442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:38.720017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:38.720017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:38.720017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:38.720017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:38.720017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:38.720017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:40.726443 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:40.727091 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:40.727091 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:40.730683 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:40.730683 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:40.730683 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:40.730683 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:40.730683 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:40.731138 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-49414b01-83df-4428-b0f5-da55f126f6ca, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:40.731138 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:40.732137 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-56b741fb-a48b-491e-9a3c-cdef706732d6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:40.732672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:40.733135 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:40.734673 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:40.734992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.736230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:40.736230 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:40.736230 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:40.736230 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:40.736230 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:40.736230 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:40.736230 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.736230 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:40.737322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.737322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:40.737322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:40.737322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:40.737322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.737322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:40.737322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:40.737322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.737322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:40.737322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:40.737322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:40.737322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.738216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:40.738216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:40.738216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:40.738216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:40.738216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:40.738216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:40.738216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:40.738216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:40.738216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:40.738216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:40.738216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:40.738216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:40.738216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:40.738625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.738625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:40.739282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.739282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:40.739282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:40.739282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:40.739282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:40.739282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:40.739282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.739282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:40.739282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:40.739282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.739282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:40.739282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:40.739282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.739282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:40.739282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:40.739282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.740345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:40.740345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:40.740345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.740345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:40.740345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:40.740345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:40.740345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.740345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:40.740345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:40.740345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:40.740345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:40.740345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:40.740345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:40.740345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:40.740345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:40.740345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:40.740345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:40.740345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:40.740345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:40.740946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:40.740946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.740946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:40.742375 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:40.742375 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:40.742542 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:40.742757 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:28:40.744786 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:40.744786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:40.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:40.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:40.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:40.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:40.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:40.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:40.747277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.747277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:40.747277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:40.747277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:40.747277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.747277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:40.747277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:40.747277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:40.747277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.747277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:40.747277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:40.747277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:40.747277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:40.747277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:40.747277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:40.748115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:40.748115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:40.748115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:40.748115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:40.748115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:40.748115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:40.748115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:40.748115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.748115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:40.748115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.748115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:40.748115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:40.748115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:40.748115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.748818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:40.748818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:40.748818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:28:40.748818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:40.748818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:40.748818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.748818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:40.748818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:40.748818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:40.748818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.748818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:40.748818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:40.748818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:40.749209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:40.749209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:40.749209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:40.749209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:40.749209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:40.749209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:40.749209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:40.749209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:40.749209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:40.749209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:40.749209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.749209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:40.750199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.750199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:40.750199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:40.750199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:40.750199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:40.750199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:40.750199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.750199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:40.750199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:40.750199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.750199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:40.750199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:40.750199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.750199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:40.750199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:40.750199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.750840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:40.750840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:40.750840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:28:40.750840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:40.750840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:40.750840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.750840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:40.750840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:40.750840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:40.750840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.750840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:40.750840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:40.750840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:40.750840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:40.750840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:40.750840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:40.750840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:40.750840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:40.751714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:40.751714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:40.751714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:40.751714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:40.751714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:40.751714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.751714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:40.754937 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:40.755605 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:40.756568 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:40.756801 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:40.758310 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:40.758310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.758647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:40.758647 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:40.758647 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:40.758647 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:40.758647 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:40.758647 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:40.758647 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.758647 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:40.760832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.760832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:40.760832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:40.760832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:40.760832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.760832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:40.760832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:40.760832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:40.760832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.760832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:40.760832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:40.760832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:40.760832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:40.760832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:40.760832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:40.761344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:40.761344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:40.761344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:40.761344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:40.761344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:40.761344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:40.761344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:40.761344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.761344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:40.762640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.762640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:40.762640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:40.762640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:40.762640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.762640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:40.762640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:40.762640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:40.762640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:40.762640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.762640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:40.763084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:40.763084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:40.763084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.763084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:40.763084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:40.763084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:40.763084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:40.763084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:40.763084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:40.763084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:40.763084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:40.763084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:40.763084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:40.763084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:40.763755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:40.763755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:40.763755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.763755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:40.764733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.764733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:40.764733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:40.764733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:40.764733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:40.764733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:40.764733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.764733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:40.764733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:40.764733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.764733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:40.764733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:40.764733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.764733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:40.764733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:40.764733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.764733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:40.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:40.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:40.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:40.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:40.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:40.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:40.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:40.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:40.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:40.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:40.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:40.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:40.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:40.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:40.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:40.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:40.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:40.766371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:40.766371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:40.766371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.766371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:40.766371 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:40.766593 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:40.766689 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:40.766967 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:40.768005 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:40.768005 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:40.768766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.769597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:40.769597 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:40.769597 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:40.769597 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:40.769597 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:40.769597 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:40.769597 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.769597 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:40.771371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.771371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:40.771371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:40.771371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:40.771371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.771371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:40.771371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:40.771371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.771371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:40.771371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:40.771371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:40.771371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.771745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:40.771745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:40.771745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:40.771745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:40.771745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:40.771745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:40.771745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:40.771745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:40.771745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:40.771745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:40.771745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:40.771745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:40.771745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:40.772181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.772181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:40.772181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.772181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:40.772181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:40.772181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:40.772181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:40.772181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:40.772181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.772181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:40.772181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:40.772181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.772181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:40.772181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:40.772181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.772681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:40.772681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:40.772681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:40.772681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:40.772681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:40.772681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.772681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:40.772681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:40.772681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:40.772681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:40.772681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:40.772681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:40.772681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:40.772681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:40.772681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:40.772681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:40.772681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:40.772681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:40.772681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:40.773265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:40.773265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:40.773265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:40.773265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:40.773265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:40.773265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:43.657503 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:43.657968 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:43.657968 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:43.658755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:43.695563 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:43.695826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:43.696541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:43.696541 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:43.696541 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:43.696541 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:43.696541 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:43.696541 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:43.696541 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:43.696541 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:43.697313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:43.697313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:43.697313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:43.697313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:43.697313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:43.697313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:43.697313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:43.697313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:43.697313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:43.697313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:43.697313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:43.697313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:43.697766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:43.697766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:43.697766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:43.697766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:43.697766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:43.697766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:43.697766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:43.697766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:43.697766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:43.697766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:43.697766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:43.697766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:43.697766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:43.698239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:43.698239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:43.698239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:43.698239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:43.698239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:43.698239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:43.698239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:43.698239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:43.698239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:43.698239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:43.698239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:43.698239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:43.698239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:43.698239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:43.698239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:43.698718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:43.698718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:43.698718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:43.698718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:43.698718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:43.698718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:43.698718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:43.698718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:43.698718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:43.698718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:43.698718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:43.698718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:43.698718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:43.698718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:43.698718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:43.698718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:43.698718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:43.698718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:43.698718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:43.699280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:43.699280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:43.699280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:43.699280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:43.699280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:43.699280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:45.726905 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:45.727383 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:45.727383 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:45.728064 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:45.728290 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:45.728447 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:45.728608 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:45.728759 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:45.728903 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-49414b01-83df-4428-b0f5-da55f126f6ca, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:45.729059 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:45.729199 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-56b741fb-a48b-491e-9a3c-cdef706732d6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:45.729355 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:45.729508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:45.729646 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:45.729782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.730343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:45.730343 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:45.730343 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:45.730343 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:45.730343 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:45.730343 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:45.730343 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.730343 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:45.731021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.731021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:45.731021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:45.731021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:45.731021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.731021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:45.731021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:45.731021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.731021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:45.731021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:45.731021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:45.731021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.731294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:45.731294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:45.731294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:45.731294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:45.731294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:45.731294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:45.731294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:45.731294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:45.731294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:45.731294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:45.731294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:45.731294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:45.731294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:45.731512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.731512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:45.731711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.731711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:45.731711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:45.731711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:45.731711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:45.731711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:45.731711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.731711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:45.731711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:45.731711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.731711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:45.731711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:45.731711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.731711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:45.731711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:45.731711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.732397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:45.732397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:45.732397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.732397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:45.732397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:45.732397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:45.732397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.732397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:45.732397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:45.732397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:45.732397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:45.732397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:45.732397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:45.732397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:45.732397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:45.732397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:45.732397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:45.732397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:45.732397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:45.732964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:45.732964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.732964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:45.742425 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:45.742627 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:45.742705 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:45.742798 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:28:45.743157 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:45.743387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.743874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:45.743874 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:45.743874 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:45.743874 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:45.743874 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:45.743874 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:45.743874 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.743874 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:45.744394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.744394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:45.744394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:45.744394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:45.744394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.744394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:45.744394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:45.744394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:45.744394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.744394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:45.744394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:45.744394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:45.744394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:45.744394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:45.744394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:45.744671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:45.744671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:45.744671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:45.744671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:45.744671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:45.744671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:45.744671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:45.744671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.744671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:45.745038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.745038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:45.745038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:45.745038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:45.745038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.745038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:45.745038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:45.745038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:28:45.745038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:45.745038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:45.745038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.745038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:45.745542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:45.745542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:45.745542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.745542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:45.745542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:45.745542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:45.745542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:45.745542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:45.745542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:45.745542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:45.745542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:45.745542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:45.745542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:45.745542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:45.746106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:45.746106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:45.746106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.746106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:45.746106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.746106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:45.746106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:45.746106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:45.746106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:45.746106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:45.746106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.746106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:45.746106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:45.746106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.746376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:45.746376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:45.746376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.746376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:45.746376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:45.746376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.746376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:45.746376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:45.746376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:28:45.746376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:45.746376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:45.746376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.746376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:45.746376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:45.746376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:45.746376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.746376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:45.746376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:45.746376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:45.746673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:45.746673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:45.746673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:45.746673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:45.746673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:45.746673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:45.746673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:45.746673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:45.746673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:45.746673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:45.746673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.746673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:45.755658 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:45.755852 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:45.755928 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:45.756089 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:45.756382 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:45.756637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.757143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:45.757143 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:45.757143 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:45.757143 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:45.757143 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:45.757143 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:45.757143 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.757143 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:45.757653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.757653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:45.757653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:45.757653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:45.757653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.757653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:45.757653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:45.757653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:45.757653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.757653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:45.757653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:45.757653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:45.757653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:45.757653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:45.757653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:45.758043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:45.758043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:45.758043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:45.758043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:45.758043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:45.758043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:45.758043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:45.758043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.758043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:45.758276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.758276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:45.758276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:45.758276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:45.758276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.758276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:45.758276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:45.758276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:45.758276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:45.758276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.758276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:45.758768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:45.758768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:45.758768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.758768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:45.758768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:45.758768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:45.758768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:45.758768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:45.758768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:45.758768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:45.758768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:45.758768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:45.758768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:45.758768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:45.759057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:45.759057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:45.759057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.759057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:45.759057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.759057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:45.759057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:45.759057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:45.759057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:45.759057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:45.759057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.759057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:45.759057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:45.759057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.759303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:45.759303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:45.759303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.759303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:45.759303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:45.759303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.759303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:45.759303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:45.759303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:45.759303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:45.759303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.759303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:45.759303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:45.759303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:45.759303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.759303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:45.759303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:45.759303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:45.759303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:45.759641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:45.759641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:45.759641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:45.759641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:45.759641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:45.759641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:45.759641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:45.759641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:45.759641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:45.759641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.759641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:45.766583 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:45.766803 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:45.766901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:45.767038 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:45.767412 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:45.767674 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:45.768068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.768590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:45.768590 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:45.768590 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:45.768590 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:45.768590 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:45.768590 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:45.768590 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.768590 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:45.769299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.769299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:45.769299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:45.769299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:45.769299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.769299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:45.769299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:45.769299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.769299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:45.769299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:45.769299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:45.769299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.769558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:45.769558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:45.769558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:45.769558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:45.769558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:45.769558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:45.769558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:45.769558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:45.769558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:45.769558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:45.769558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:45.769558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:45.769558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:45.769761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.769761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:45.769969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.769969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:45.769969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:45.769969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:45.769969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:45.769969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:45.769969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.769969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:45.769969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:45.769969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.769969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:45.769969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:45.769969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.769969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:45.769969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:45.769969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:45.770595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:45.770595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:45.770595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.770595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:45.770595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:45.770595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:45.770595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:45.770595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:45.770595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:45.770595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:45.770595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:45.770595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:45.770595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:45.770595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:45.770595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:45.770595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:45.770595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:45.770595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:45.770595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:45.771136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:45.771136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:45.771136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:48.659290 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:48.659290 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:48.659290 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:48.659290 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:48.712271 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:48.712544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:48.713202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:48.713202 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:48.713202 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:48.713202 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:48.713202 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:48.713202 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:48.713202 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:48.713202 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:48.713983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:48.713983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:48.713983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:48.713983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:48.713983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:48.713983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:48.713983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:48.713983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:48.713983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:48.713983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:48.713983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:48.713983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:48.714530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:48.714530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:48.714530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:48.714530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:48.714530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:48.714530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:48.714530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:48.714530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:48.714530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:48.714530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:48.714530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:48.714530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:48.714530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:48.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:48.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:48.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:48.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:48.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:48.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:48.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:48.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:48.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:48.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:48.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:48.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:48.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:48.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:48.715077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:48.715678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:48.715678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:48.715678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:48.715678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:48.715678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:48.715678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:48.715678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:48.715678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:48.715678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:48.715678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:48.715678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:48.715678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:48.715678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:48.715678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:48.715678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:48.715678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:48.715678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:48.715678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:48.715678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:48.716021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:48.716021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:48.716021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:48.716021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:48.716021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:48.716021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:50.727167 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:50.727636 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:50.727636 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:50.728597 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:50.728830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:50.729037 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:50.729231 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:50.729418 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:50.729591 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-49414b01-83df-4428-b0f5-da55f126f6ca, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:50.729737 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:50.729895 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-56b741fb-a48b-491e-9a3c-cdef706732d6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:50.730061 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:50.730225 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:50.730407 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:50.730553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.731158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:50.731158 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:50.731158 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:50.731158 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:50.731158 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:50.731158 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:50.731158 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.731158 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:50.731858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.731858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:50.731858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:50.731858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:50.731858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.731858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:50.731858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:50.731858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.731858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:50.731858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:50.731858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:50.731858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.732318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:50.732318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:50.732318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:50.732318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:50.732318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:50.732318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:50.732318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:50.732318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:50.732318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:50.732318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:50.732318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:50.732318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:50.732318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:50.732736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.732736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:50.732736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.732736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:50.732736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:50.732736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:50.732736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:50.732736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:50.732736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.732736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:50.732736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:50.732736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.732736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:50.732736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:50.732736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.733247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:50.733247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:50.733247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.733247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:50.733247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:50.733247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.733247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:50.733247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:50.733247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:50.733247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.733247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:50.733247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:50.733247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:50.733247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:50.733247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:50.733247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:50.733247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:50.733247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:50.733247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:50.733873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:50.733873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:50.733873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:50.733873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:50.733873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.733873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:50.742684 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:50.742859 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:50.742933 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:50.743042 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:28:50.743551 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:50.743876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.744724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:50.744724 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:50.744724 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:50.744724 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:50.744724 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:50.744724 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:50.744724 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.744724 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:50.745264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.745264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:50.745264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:50.745264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:50.745264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.745264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:50.745264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:50.745264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:50.745264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.745264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:50.745264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:50.745264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:50.745264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:50.745264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:50.745264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:50.745816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:50.745816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:50.745816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:50.745816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:50.745816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:50.745816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:50.745816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:50.745816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.745816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:50.745816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.745816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:50.745816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:50.745816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:50.745816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.746341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:50.746341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:50.746341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:28:50.746341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:50.746341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:50.746341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.746341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:50.746341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:50.746341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:50.746341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.746341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:50.746341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:50.746341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:50.746779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:50.746779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:50.746779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:50.746779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:50.746779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:50.746779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:50.746779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:50.746779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:50.746779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:50.746779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:50.746779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.746779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:50.747185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.747185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:50.747185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:50.747185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:50.747185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:50.747185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:50.747185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.747185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:50.747185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:50.747185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.747185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:50.747185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:50.747185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.747185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:50.747185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:50.747185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:50.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:50.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:28:50.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:50.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:50.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:50.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:50.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:50.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:50.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:50.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:50.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:50.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:50.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:50.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:50.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:50.752368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:50.752368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:50.752368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:50.752368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:50.752368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:50.752368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.752368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:50.755946 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:50.756139 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:50.756223 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:50.756341 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:50.756747 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:50.756947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.757380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:50.757380 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:50.757380 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:50.757380 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:50.757380 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:50.757380 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:50.757380 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.757380 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:50.757942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.757942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:50.757942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:50.757942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:50.757942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.757942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:50.757942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:50.757942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:50.757942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.757942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:50.757942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:50.757942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:50.757942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:50.757942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:50.757942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:50.758579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:50.758579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:50.758579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:50.758579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:50.758579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:50.758579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:50.758579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:50.758579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.758579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:50.758579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.758579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:50.758579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:50.758579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:50.758579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.759119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:50.759119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:50.759119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:50.759119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:50.759119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.759119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:50.759119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:50.759119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:50.759119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.759119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:50.759119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:50.759119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:50.759119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:50.759587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:50.759587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:50.759587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:50.759587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:50.759587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:50.759587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:50.759587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:50.759587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:50.759587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:50.759587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.759587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:50.759587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.760037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:50.760037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:50.760037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:50.760037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:50.760037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:50.760037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.760037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:50.760037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:50.760037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.760037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:50.760037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:50.760037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.760037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:50.760037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:50.760037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.760037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:50.760037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:50.760037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:50.760037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.760748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:50.766833 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:50.766990 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:50.767082 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:50.767205 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:50.767713 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:50.767996 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:50.768444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.769021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:50.769021 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:50.769021 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:50.769021 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:50.769021 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:50.769021 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:50.769021 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.769021 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:50.771398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.771398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:50.771398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:50.771398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:50.771398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.771398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:50.771398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:50.771398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.771398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:50.771398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:50.771398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:50.771398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.771627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:50.771627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:50.771627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:50.771627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:50.771627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:50.771627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:50.771627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:50.771627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:50.771627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:50.771627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:50.771627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:50.771627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:50.771627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:50.771832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.771832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:50.771832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.771832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:50.771832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:50.771832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:50.771832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:50.771832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:50.771832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.771832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:50.771832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:50.771832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.771832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:50.771832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:50.771832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.772113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:50.772113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:50.772113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:50.772113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:50.772113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:50.772113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.772113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:50.772113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:50.772113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:50.772113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:50.772113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:50.772113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:50.772113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:50.772113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:50.772113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:50.772113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:50.772113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:50.772113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:50.772113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:50.772557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:50.772557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:50.772557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:50.772557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:50.772557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:50.772557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:53.658281 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:53.660862 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:53.660862 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:53.660862 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:53.678670 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=245 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:28:53.701094 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:53.701229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:53.703479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:53.703479 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:53.703479 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:53.703479 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:53.703479 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:53.703479 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:53.703479 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:53.703479 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:53.703479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:53.703479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:53.703479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:53.703479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:53.703479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:53.703479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:53.703479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:53.703965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:53.703965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:53.703965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:53.703965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:53.703965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:53.703965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:53.703965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:53.703965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:53.703965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:53.703965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:53.703965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:53.703965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:53.703965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:53.703965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:53.704269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:53.704269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:53.704269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:53.704269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:53.704269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:53.704269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:53.704269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:53.704269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:53.704269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:53.704269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:53.704269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:53.704269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:53.704269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:53.704269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:53.704566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:53.704891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:53.704891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:53.704891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:53.704891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:53.704891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:53.704891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:53.704891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:53.704891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:53.704891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:55.728647 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:55.730531 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:55.730626 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:55.731609 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:55.731800 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:55.731895 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:55.732006 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:55.732141 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:55.732295 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-49414b01-83df-4428-b0f5-da55f126f6ca, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:55.732404 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:55.733203 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-56b741fb-a48b-491e-9a3c-cdef706732d6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:55.733356 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:55.733470 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:55.733599 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:55.733817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.734488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:55.734488 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:55.734488 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:55.734488 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:55.734488 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:55.734488 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:55.734488 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.734488 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:55.735190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.735190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:55.735190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:55.735190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:55.735190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.735190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:55.735190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:55.735190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.735190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:55.735190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:55.735190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:55.735190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.735663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:55.735663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:55.735663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:55.735663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:55.735663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:55.735663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:55.735663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:55.735663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:55.735663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:55.735663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:55.735663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:55.735663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:55.735663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:55.736076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.736076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:55.736076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.736076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:55.736076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:55.736076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:55.736076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:55.736076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:55.736076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.736076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:55.736076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:55.736076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.736076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:55.736076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:55.736076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.736353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:55.736353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:55.736353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.736353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:28:55.736353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:55.736353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.736353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:55.736353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:55.736353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:55.736353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.736353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:55.736353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:55.736353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:55.736353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:55.736353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:55.736353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:55.736353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:55.736353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:55.736353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:55.737488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:55.737488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:55.737488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:55.737488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:55.737488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.737488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:55.743094 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:55.745540 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:55.745687 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:55.745687 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:28:55.746242 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:55.746487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.747371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:55.747371 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:55.747371 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:55.747371 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:55.747371 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:55.747371 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:55.747371 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.747371 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:55.747958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.747958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:55.747958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:55.747958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:55.747958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.747958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:55.747958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:55.747958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:55.747958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.747958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:55.747958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:55.747958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:55.747958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:55.747958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:55.747958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:55.748631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:55.748631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:55.748631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:55.748631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:55.748631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:55.748631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:55.748631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:55.748631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.748631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:55.750982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.750982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:55.750982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:55.750982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:55.750982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.750982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:55.750982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:55.750982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:28:55.750982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:55.750982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:55.750982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.750982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:55.751445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:55.751445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:55.751445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.751445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:55.751445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:55.751445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:55.751445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:55.751445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:55.751445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:55.751445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:55.751445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:55.751445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:55.751445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:55.751445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:55.751817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:55.751817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:55.751817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.751817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:55.752339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.752339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:55.752339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:55.752339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:55.752339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:55.752339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:55.752339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.752339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:55.752339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:55.752339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.752339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:55.752339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:55.752339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.752339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:55.752339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:55.752339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.753195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:28:55.753195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:28:55.753195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:28:55.753195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:55.753195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:55.753195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.753195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:55.753195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:55.753195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:55.753195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.753195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:55.753195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:55.753195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:55.753195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:55.753195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:55.753195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:55.753195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:55.753195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:55.754017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:55.754017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:55.754017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:55.754017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:55.754017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:55.754017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.754017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:55.756353 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:55.756553 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:55.756664 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:55.756817 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:55.757306 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:55.757546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.758191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:55.758191 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:55.758191 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:55.758191 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:55.758191 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:55.758191 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:55.758191 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.758191 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:55.759110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.759110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:28:55.759110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:55.759110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:55.759110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.759110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:55.759110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:55.759110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:28:55.759110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.759110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:28:55.759110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:55.759110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:28:55.759110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:55.759110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:28:55.759110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:28:55.759621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:55.759621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:28:55.759621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:55.759621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:28:55.759621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:28:55.759621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:55.759621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:28:55.759621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.759621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:28:55.760474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.760474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:55.760474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:55.760474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:55.760474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.760474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:55.760474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:55.760474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:55.760474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:55.760474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.760474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:55.761403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:55.761403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:55.761403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.761403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:55.761403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:55.761403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:55.761403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:55.761403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:55.761403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:55.761403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:55.761403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:55.761403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:55.761403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:55.761403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:55.762157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:55.762157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:55.762157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.762157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:55.762157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.762157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:55.762157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:55.762157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:55.762157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:55.762157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:55.762157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.762157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:55.762157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:55.762157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.762715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:55.762715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:55.762715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.762715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:55.762715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:55.762715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.762715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:28:55.762715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:28:55.762715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:28:55.762715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:28:55.762715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.762715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:55.762715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:55.762715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:55.762715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.762715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:55.762715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:55.762715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:55.762715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:55.763189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:55.763189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:55.763189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:55.763189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:55.763189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:55.763189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:55.763189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:55.763189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:55.763189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:55.763189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.763189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:55.767156 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:55.767406 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:55.767506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:55.767693 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:55.768152 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:55.768394 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:55.768860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.769452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:55.769452 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:55.769452 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:55.769452 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:55.769452 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:55.769452 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:55.769452 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.769452 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:55.770357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.770357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:55.770357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:55.770357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:55.770357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.770357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:55.770357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:55.770357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.770357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:55.770357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:55.770357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:55.770357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.770748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:55.770748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:55.770748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:55.770748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:55.770748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:55.770748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:55.770748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:55.770748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:55.770748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:55.770748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:55.770748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:55.770748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:55.770748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:55.771133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.771133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:55.771845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.771845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:55.771845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:55.771845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:55.771845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:55.771845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:55.771845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.771845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:55.771845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:55.771845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.771845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:55.771845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:55.771845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.771845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:55.771845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:55.771845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:55.773043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:28:55.773043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:55.773043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.773043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:55.773043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:55.773043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:55.773043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:55.773043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:55.773043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:55.773043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:55.773043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:55.773043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:55.773043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:55.773043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:55.773043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:55.773043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:55.773043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:55.773043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:55.773043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:55.774105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:55.774105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:55.774105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:28:58.659829 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:28:58.662357 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:28:58.662357 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:28:58.662357 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:28:58.701959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:28:58.702352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:58.702895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:28:58.702895 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:58.702895 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:28:58.702895 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:28:58.702895 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:58.702895 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:28:58.702895 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:58.702895 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:28:58.704615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:58.704615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:28:58.704615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:58.704615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:28:58.704615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:58.704615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:58.704615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:58.704615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:58.704615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:58.704615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:58.704615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:28:58.704615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:58.705056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:28:58.705056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:58.705056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:28:58.705056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:58.705056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:28:58.705056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:28:58.705056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:58.705056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:28:58.705056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:58.705056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:28:58.705056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:28:58.705056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:58.705056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:28:58.705375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:58.705375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:28:58.705375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:58.705375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:28:58.705375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:28:58.705375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:28:58.705375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:28:58.705375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:28:58.705375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:58.705375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:28:58.705375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:28:58.705375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:58.705375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:28:58.705375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:28:58.705375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:28:58.705731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:28:58.705731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:28:58.705731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:28:58.705731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:28:58.705731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:28:58.705731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:58.705731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:58.705731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:28:58.705731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:28:58.705731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:28:58.705731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:28:58.705731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:28:58.705731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:28:58.705731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:28:58.705731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:28:58.705731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:28:58.705731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:28:58.705731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:28:58.705731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:28:58.706124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:28:58.706124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:28:58.706124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:28:58.706124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:28:58.706124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:28:58.706124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:00.730584 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:00.731403 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:00.731614 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:00.735481 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:00.735758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:00.735953 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:00.736208 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:00.736421 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-49414b01-83df-4428-b0f5-da55f126f6ca, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:00.736668 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:00.736855 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-56b741fb-a48b-491e-9a3c-cdef706732d6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:00.737074 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:00.737275 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:00.737462 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:00.737631 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:00.737789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.738608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:00.738608 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:00.738608 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:00.738608 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:00.738608 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:00.738608 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:00.738608 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.738608 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:00.739460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.739460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:00.739460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:00.739460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:00.739460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.739460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:00.739460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:00.739460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.739460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:00.739460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:00.739460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:00.739460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.739742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:00.739742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:00.739742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:00.739742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:00.739742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:00.739742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:00.739742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:00.739742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:00.739742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:00.739742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:00.739742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:00.739742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:00.739742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:00.739972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.739972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:00.743071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.743071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:00.743071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:00.743071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:00.743071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:00.743071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:00.743071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.743071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:00.743071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:00.743071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.743071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:00.743071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:00.743071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.743071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:00.743071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:00.743071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.743451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:00.743451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:00.743451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.743451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:00.743451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:00.743451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:00.743451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.743451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:00.743451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:00.743451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:00.743451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:00.743451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:00.743451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:00.743451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:00.743451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:00.743451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:00.743451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:00.743451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:00.743451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:00.743829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:00.743829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.743829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:00.746315 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:00.746315 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:00.746315 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:00.746315 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:29:00.746315 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:00.746315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.746613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:00.746613 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:00.746613 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:00.746613 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:00.746613 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:00.746613 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:00.746613 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.746613 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:00.747337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.747337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:00.747337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:00.747337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:00.747337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.747337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:00.747337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:00.747337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:00.747337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.747337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:00.747337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:00.747337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:00.747337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:00.747337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:00.747337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:00.748525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.748525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:00.748525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:00.748525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:00.748525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.748525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:00.748525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:00.748525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:29:00.748525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:00.748525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:00.748525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.748525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:00.748799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:00.748799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:00.748799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.748799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:00.748799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:00.748799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:00.748799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:00.748799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:00.748799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:00.748799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:00.748799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:00.748799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:00.748799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:00.748799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:00.749072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:00.749072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:00.749072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.749072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:00.751201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.751201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:00.751201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:00.751201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:00.751201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:00.751201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:00.751201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.751201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:00.751201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:00.751201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.751201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:00.751201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:00.751201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.751201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:00.751201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:00.751201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.751564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:00.751564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:00.751564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:29:00.751564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:00.751564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:00.751564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.751564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:00.751564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:00.751564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:00.751564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.751564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:00.751564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:00.751564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:00.751564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:00.751564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:00.751564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:00.751564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:00.751564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:00.751913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:00.751913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:00.751913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:00.751913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:00.751913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:00.751913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.751913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:00.757048 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:00.757048 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:00.757048 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:00.757048 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:00.757417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:00.757681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.758097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:00.758097 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:00.758097 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:00.758097 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:00.758097 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:00.758097 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:00.758097 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.758097 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:00.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:00.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:00.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:00.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:00.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:00.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:00.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:00.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:00.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:00.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:00.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:00.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:00.760533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:00.760533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:00.760533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:00.760533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:00.760533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:00.760533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:00.760533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:00.760533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.760533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:00.760533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.760533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:00.760533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:00.760533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:00.760533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.760771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:00.760771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:00.760771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:00.760771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:00.760771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.760771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:00.760771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:00.760771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:00.760771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.760771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:00.760771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:00.760771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:00.760771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:00.761183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:00.761183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:00.761183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:00.761183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:00.761183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:00.761183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:00.761183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:00.761183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:00.761183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:00.761183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.761183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:00.761183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.761408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:00.761408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:00.761408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:00.761408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:00.761408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:00.761408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.761408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:00.761408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:00.761408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.761408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:00.761408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:00.761408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.761408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:00.761408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:00.761408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.761408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:00.761408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:00.761408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:00.761408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.761762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:00.767450 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:00.767761 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:00.767959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:00.768185 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:00.768558 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:00.768859 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:00.769225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.771518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:00.771518 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:00.771518 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:00.771518 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:00.771518 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:00.771518 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:00.771518 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.771518 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:00.773460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.773460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:00.773460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:00.773460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:00.773460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.773460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:00.773460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:00.773460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.773460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:00.773460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:00.773460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:00.773460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.773787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:00.773787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:00.773787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:00.773787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:00.773787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:00.773787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:00.773787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:00.773787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:00.773787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:00.773787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:00.773787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:00.773787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:00.773787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:00.773990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.773990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:00.775487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.775487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:00.775487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:00.775487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:00.775487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:00.775487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:00.775487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.775487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:00.775487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:00.775487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.775487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:00.775487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:00.775487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.775487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:00.775487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:00.775487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:00.775882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:00.775882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:00.775882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.775882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:00.775882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:00.775882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:00.775882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:00.775882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:00.775882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:00.775882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:00.775882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:00.775882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:00.775882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:00.775882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:00.775882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:00.775882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:00.775882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:00.775882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:00.775882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:00.776287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:00.776287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:00.776287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:03.661118 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:03.663574 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:03.663574 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:03.663574 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:03.701614 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:03.702204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:03.702857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:03.702857 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:03.702857 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:03.702857 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:03.702857 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:03.702857 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:03.702857 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:03.702857 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:03.704236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:03.704236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:03.704236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:03.704236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:03.704236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:03.704236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:03.704236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:03.704236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:03.704236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:03.704236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:03.704236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:03.704236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:03.704791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:03.704791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:03.704791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:03.704791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:03.704791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:03.704791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:03.704791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:03.704791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:03.704791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:03.704791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:03.704791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:03.704791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:03.704791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:03.705181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:03.705181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:03.706539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:03.706539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:03.706539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:03.706539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:03.706539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:03.706539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:03.706539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:03.706539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:03.706539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:03.706539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:03.706539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:03.706539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:03.706539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:03.706539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:03.706539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:03.706539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:03.707107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:03.707107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:03.707107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:03.707107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:03.707107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:03.707107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:03.707107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:03.707107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:03.707107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:03.707107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:03.707107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:03.707107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:03.707107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:03.707107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:03.707107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:03.707107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:03.707107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:03.707107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:03.707107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:03.707675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:03.707675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:03.707675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:05.731510 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:05.731997 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:05.731997 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:05.733379 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:05.733379 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:05.733502 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:05.734148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:05.734148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-49414b01-83df-4428-b0f5-da55f126f6ca, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:05.734148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:05.734148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-56b741fb-a48b-491e-9a3c-cdef706732d6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:05.734148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:05.734368 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:05.734368 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:05.734368 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:05.734467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.734885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:05.734885 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:05.734885 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:05.734885 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:05.734885 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:05.734885 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:05.734885 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.734885 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:05.737332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.737332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:05.737332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:05.737332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:05.737332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.737332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:05.737332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:05.737332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.737332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:05.737332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:05.737332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:05.737332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.737750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:05.737750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:05.737750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:05.737750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:05.737750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:05.737750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:05.737750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:05.737750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:05.737750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:05.737750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:05.737750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:05.737750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:05.737750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:05.738004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.738004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:05.738004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.738004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:05.738004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:05.738004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:05.738004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:05.738004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:05.738004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.738004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:05.738004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:05.738004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.738004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:05.738004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:05.738004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.738295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:05.738295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:05.738295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.738295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:05.738295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:05.738295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.738295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:05.738295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:05.738295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:05.738295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.738295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:05.738295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:05.738295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:05.738295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:05.738295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:05.738295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:05.738295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:05.738295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:05.738295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:05.738651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:05.738651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:05.738651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:05.738651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:05.738651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.738651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:05.745594 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:05.745751 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:05.745832 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:05.745917 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:29:05.746287 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:05.746489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.746927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:05.746927 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:05.746927 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:05.746927 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:05.746927 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:05.746927 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:05.746927 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.746927 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:05.747473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.747473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:05.747473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:05.747473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:05.747473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.747473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:05.747473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:05.747473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:05.747473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.747473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:05.747473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:05.747473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:05.747473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:05.747473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:05.747473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:05.747832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:05.747832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:05.747832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:05.747832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:05.747832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:05.747832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:05.747832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:05.747832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.747832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:05.748018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.748018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:05.748018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:05.748018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:05.748018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.748018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:05.748018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:05.748018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:29:05.748018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:05.748018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:05.748018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.748018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:05.748274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:05.748274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:05.748274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.748274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:05.748274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:05.748274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:05.748274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:05.748274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:05.748274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:05.748274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:05.748274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:05.748274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:05.748274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:05.748274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:05.748536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:05.748536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:05.748536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.748536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:05.748663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.748663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:05.748663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:05.748663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:05.748663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:05.748663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:05.748663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.748663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:05.748663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:05.748663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.748663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:05.748663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:05.748663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.748663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:05.748663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:05.748663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.749361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:05.749361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:05.749361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:29:05.749361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:05.749361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:05.749361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.749361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:05.749361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:05.749361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:05.749361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.749361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:05.749361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:05.749361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:05.749361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:05.749361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:05.749361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:05.749361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:05.749361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:05.749703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:05.749703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:05.749703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:05.749703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:05.749703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:05.749703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.749703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:05.756812 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:05.756959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:05.757064 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:05.757157 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:05.757576 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:05.757778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.758249 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:05.758249 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:05.758249 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:05.758249 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:05.758249 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:05.758249 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:05.758249 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.758249 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:05.758777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.758777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:05.758777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:05.758777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:05.758777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.758777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:05.758777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:05.758777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:05.758777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.758777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:05.758777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:05.758777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:05.758777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:05.758777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:05.758777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:05.759278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:05.759278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:05.759278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:05.759278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:05.759278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:05.759278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:05.759278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:05.759278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.759278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:05.759518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.759518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:05.759518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:05.759518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:05.759518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.759518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:05.759518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:05.759518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:05.759518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:05.759518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.759518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:05.759710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:05.759710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:05.759710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.759710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:05.759710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:05.759710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:05.759710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:05.759710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:05.759710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:05.759710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:05.759710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:05.759710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:05.759710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:05.759710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:05.759965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:05.759965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:05.759965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.759965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:05.760095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.760095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:05.760095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:05.760095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:05.760095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:05.760095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:05.760095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.760095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:05.760095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:05.760095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.760095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:05.760095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:05.760095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.760095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:05.760095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:05.760095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.760095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:05.760396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:05.760396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:05.760396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:05.760396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.760396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:05.760396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:05.760396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:05.760396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.760396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:05.760396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:05.760396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:05.760396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:05.760396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:05.760396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:05.760396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:05.760396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:05.760396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:05.760396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:05.760396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:05.760705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:05.760705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:05.760705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.760705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:05.767800 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:05.767950 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:05.767994 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:05.768106 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:05.768503 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:05.768702 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:05.768957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.769402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:05.769402 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:05.769402 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:05.769402 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:05.769402 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:05.769402 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:05.769402 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.769402 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:05.769974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.769974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:05.769974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:05.769974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:05.769974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.769974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:05.769974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:05.769974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.769974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:05.769974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:05.769974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:05.769974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.770325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:05.770325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:05.770325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:05.770325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:05.770325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:05.770325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:05.770325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:05.770325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:05.770325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:05.770325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:05.770325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:05.770325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:05.770325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:05.770552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.770552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:05.770610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.770610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:05.770610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:05.770610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:05.770610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:05.770610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:05.770610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.770610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:05.770610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:05.770610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.770610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:05.770610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:05.770610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.770610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:05.770610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:05.770610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:05.770924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:05.770924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:05.770924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.770924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:05.770924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:05.770924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:05.770924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:05.770924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:05.770924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:05.770924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:05.770924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:05.770924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:05.770924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:05.770924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:05.770924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:05.770924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:05.770924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:05.770924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:05.770924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:05.771343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:05.771343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:05.771343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:08.662011 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:08.662478 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:08.662478 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:08.663137 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:08.716274 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:08.716274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:08.716274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:08.716274 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:08.716274 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:08.716274 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:08.716274 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:08.716274 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:08.716274 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:08.716274 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:08.716274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.051 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:08.716274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:08.716274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:08.716692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:08.716692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:08.716692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:08.716692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:08.716692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:08.716692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:08.716692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:08.716692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:08.716692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:08.716692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:08.716692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:08.716692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:08.716692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:08.716692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:08.716971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:08.716971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:08.716971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:08.716971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:08.716971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:08.716971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:08.716971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:08.716971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:08.716971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:08.716971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:08.716971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:08.716971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:08.717229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:08.717229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:08.717229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:08.717229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:08.717229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:08.717229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:08.717229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:08.717229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:08.717229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:08.717229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:08.717229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:08.717229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:08.717229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:08.717229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:08.717229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:08.717229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:08.717229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:08.717229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:08.717229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:08.717632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:08.717632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:08.717632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:08.717632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:08.717632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:08.717632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:08.717632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:08.717632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:08.717632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:08.717632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:08.717632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:08.717632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:08.717632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:08.717632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:08.717632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:08.717632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:08.717632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:10.731958 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:10.732599 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:10.732599 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:10.735807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:10.735807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:10.735807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:10.735807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:10.735807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:10.736220 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:10.736220 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:10.736220 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:10.736220 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:10.736220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.736220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:10.736220 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:10.736220 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:10.736220 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:10.736761 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:10.736761 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:10.736761 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.736761 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:10.736982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.736982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:10.736982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:10.736982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:10.736982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.736982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:10.736982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:10.736982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.736982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:10.736982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:10.736982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:10.736982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.737695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:10.737695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:10.737695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:10.737695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:10.737695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:10.737695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:10.737695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:10.737695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:10.737695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:10.737695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:10.737695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:10.737695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:10.737695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:10.738480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.738480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:10.738480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.738480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:10.738480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:10.738480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:10.738480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:10.738480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:10.738480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.738480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:10.738480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:10.738480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.738480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:10.738480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:10.738480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.739010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:10.739010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:10.739010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.739010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:10.739010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:10.739010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.739010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:10.739010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:10.739010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:10.739010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.739010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:10.739010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:10.739010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:10.739010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:10.739010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:10.739010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:10.739010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:10.739010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:10.739010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:10.739701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:10.739701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:10.739701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:10.739701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:10.739701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.739701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:10.745925 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:10.746181 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:10.746315 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:10.746456 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:29:10.747206 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:10.747206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.747423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:10.747423 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:10.747423 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:10.747423 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:10.747423 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:10.747423 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:10.747423 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.747423 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:10.748081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.748081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:10.748081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:10.748081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:10.748081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.748081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:10.748081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:10.748081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:10.748081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.748081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:10.748081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:10.748081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:10.748081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:10.748081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:10.748081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:10.749006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:10.749006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:10.749006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:10.749006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:10.749006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:10.749006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:10.749006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:10.749006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.749006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:10.749554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.749554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:10.749554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:10.749554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:10.749554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.749554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:10.749554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:10.749554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:29:10.749554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:10.749554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:10.749554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.749554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:10.750294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:10.750294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:10.750294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.750294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:10.750294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:10.750294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:10.750294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:10.750294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:10.750294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:10.750294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:10.750294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:10.750294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:10.750294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:10.750294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:10.750687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:10.750687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:10.750687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.750687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:10.753313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.753313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:10.753313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:10.753313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:10.753313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:10.753313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:10.753313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.753313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:10.753313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:10.753313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.753313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:10.753313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:10.753313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.753313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:10.753313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:10.753313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.753849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:10.753849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:10.753849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:29:10.753849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:10.753849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:10.753849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.753849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:10.753849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:10.753849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:10.753849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.753849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:10.753849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:10.753849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:10.753849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:10.753849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:10.753849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:10.753849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:10.753849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:10.754443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:10.754443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:10.754443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:10.754443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:10.754443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:10.754443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.754443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:10.757180 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:10.760068 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:10.760195 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:10.760355 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:10.761242 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:10.761242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.762280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:10.762280 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:10.762280 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:10.762280 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:10.762280 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:10.762280 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:10.762280 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.762280 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:10.763116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.763116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:10.763116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:10.763116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:10.763116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.763116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:10.763116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:10.763116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:10.763116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.763116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:10.763116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:10.763116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:10.763116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:10.763116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:10.763116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:10.763930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:10.763930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:10.763930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:10.763930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:10.763930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:10.763930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:10.763930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:10.763930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.763930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:10.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:10.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:10.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:10.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:10.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:10.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:10.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:10.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.764509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:10.765131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:10.765131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:10.765131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.765131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:10.765131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:10.765131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:10.765131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:10.765131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:10.765131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:10.765131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:10.765131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:10.765131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:10.765131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:10.765131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:10.765603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:10.765603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:10.765603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.765603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:10.766615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.766615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:10.766615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:10.766615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:10.766615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:10.766615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:10.766615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.766615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:10.766615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:10.766615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.766615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:10.766615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:10.766615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.766615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:10.766615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:10.766615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.766615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:10.767091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:10.767091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:10.767091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:10.767091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.767091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:10.767091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:10.767091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:10.767091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.767091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:10.767091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:10.767091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:10.767091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:10.767091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:10.767091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:10.767091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:10.767091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:10.767091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:10.767091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:10.767091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:10.767605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:10.767605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:10.767605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.767605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:10.769277 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:10.769506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:10.769629 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:10.769808 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:10.771132 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:10.771132 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:10.771132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.772345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:10.772345 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:10.772345 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:10.772345 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:10.772345 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:10.772345 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:10.772345 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.772345 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:10.773141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.773141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:10.773141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:10.773141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:10.773141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.773141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:10.773141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:10.773141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.773141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:10.773141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:10.773141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:10.773141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.773576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:10.773576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:10.773576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:10.773576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:10.773576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:10.773576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:10.773576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:10.773576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:10.773576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:10.773576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:10.773576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:10.773576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:10.773576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:10.773944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.773944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:10.774330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.774330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:10.774330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:10.774330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:10.774330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:10.774330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:10.774330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.774330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:10.774330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:10.774330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.774330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:10.774330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:10.774330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.774330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:10.774330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:10.774330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:10.774789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:10.774789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:10.774789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.774789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:10.774789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:10.774789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:10.774789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:10.774789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:10.774789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:10.774789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:10.774789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:10.774789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:10.774789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:10.774789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:10.774789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:10.774789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:10.774789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:10.774789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:10.774789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:10.775318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:10.775318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:10.775318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:13.662410 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:13.662834 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:13.662834 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:13.663369 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:13.702375 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:13.703283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:13.703436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:13.703436 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:13.703436 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:13.703436 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:13.703436 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:13.703436 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:13.703436 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:13.703436 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:13.704125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:13.704125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:13.704125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:13.704125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:13.704125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:13.704125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:13.704125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:13.704125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:13.704125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:13.704125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:13.704125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:13.704125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:13.704532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:13.704532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:13.704532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:13.704532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:13.704532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:13.704532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:13.704532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:13.704532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:13.704532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:13.704532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:13.704532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:13.704532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:13.704532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:13.704881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:13.704881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:13.704881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:13.704881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:13.704881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:13.704881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:13.704881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:13.704881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:13.704881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:13.704881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:13.704881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:13.704881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:13.704881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:13.704881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:13.704881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:13.705366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:13.705366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:13.705366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:13.705366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:13.705366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:13.705366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:13.705366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:13.705366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:13.705366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:13.705366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:13.705366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:13.705366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:13.705366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:13.705366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:13.705366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:13.705366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:13.705366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:13.705366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:13.705366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:13.705873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:13.705873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:13.705873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:13.705873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:13.705873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:13.705873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:15.732371 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:15.732786 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:15.732786 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:15.733472 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:15.733606 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:15.733708 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:15.733810 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:15.733901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:15.733993 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:15.734099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:15.734190 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:15.734318 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:15.734423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.734803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:15.734803 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:15.734803 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:15.734803 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:15.734803 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:15.734803 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:15.734803 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.734803 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:15.735562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.735562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:15.735562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:15.735562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:15.735562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.735562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:15.735562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:15.735562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.735562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:15.735562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:15.735562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:15.735562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.736163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:15.736163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:15.736163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:15.736163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:15.736163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:15.736163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:15.736163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:15.736163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:15.736163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:15.736163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:15.736163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:15.736163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:15.736163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:15.736628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.736628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:15.736703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.736703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:15.736703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:15.736703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:15.736703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:15.736703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:15.736703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.736703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:15.736703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:15.736703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.736703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:15.736703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:15.736703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.736703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:15.736703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:15.736703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.737226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:15.737226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:15.737226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.737226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:15.737226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:15.737226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:15.737226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.737226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:15.737226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:15.737226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:15.737226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:15.737226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:15.737226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:15.737226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:15.737226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:15.737226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:15.737226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:15.737226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:15.737226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:15.737670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:15.737670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.737670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:15.746163 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:15.746366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:15.746453 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:15.746576 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:29:15.747110 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:15.747261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.747883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:15.747883 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:15.747883 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:15.747883 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:15.747883 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:15.747883 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:15.747883 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.747883 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:15.748407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.748407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:15.748407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:15.748407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:15.748407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.748407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:15.748407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:15.748407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:15.748407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.748407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:15.748407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:15.748407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:15.748407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:15.748407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:15.748407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:15.748702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:15.748702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:15.748702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:15.748702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:15.748702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:15.748702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:15.748702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:15.748702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.748702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:15.748969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.748969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:15.748969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:15.748969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:15.748969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.748969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:15.748969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:15.748969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:29:15.748969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:15.748969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:15.748969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.748969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:15.749409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:15.749409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:15.749409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.749409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:15.749409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:15.749409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:15.749409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:15.749409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:15.749409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:15.749409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:15.749409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:15.749409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:15.749409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:15.749409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:15.749905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:15.749905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:15.749905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.749905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:15.750050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.750050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:15.750050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:15.750050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:15.750050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:15.750050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:15.750050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.750050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:15.750050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:15.750050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.750050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:15.750050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:15.750050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.750050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:15.750050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:15.750050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.750613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:15.750613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:15.750613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:29:15.750613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:15.750613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:15.750613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.750613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:15.750613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:15.750613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:15.750613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.750613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:15.750613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:15.750613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:15.750613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:15.750613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:15.750613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:15.750613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:15.750613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:15.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:15.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:15.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:15.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:15.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:15.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:15.760088 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:15.760302 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:15.760425 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:15.760516 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:15.761310 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:15.761491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.761879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:15.761879 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:15.761879 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:15.761879 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:15.761879 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:15.761879 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:15.761879 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.761879 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:15.762444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.762444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:15.762444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:15.762444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:15.762444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.762444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:15.762444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:15.762444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:15.762444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.762444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:15.762444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:15.762444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:15.762444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:15.762444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:15.762444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:15.762842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:15.762842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:15.762842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:15.762842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:15.762842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:15.762842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:15.762842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:15.762842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.762842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:15.763077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.763077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:15.763077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:15.763077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:15.763077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.763077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:15.763077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:15.763077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:15.763077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:15.763077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.763077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:15.763418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:15.763418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:15.763418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.763418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:15.763418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:15.763418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:15.763418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:15.763418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:15.763418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:15.763418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:15.763418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:15.763418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:15.763418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:15.763418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:15.763783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:15.763783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:15.763783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.763783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:15.763783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.763783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:15.763783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:15.763783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:15.763783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:15.763783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:15.763783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.763783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:15.763783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:15.763783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.764180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:15.764180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:15.764180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.764180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:15.764180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:15.764180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.764180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:15.764180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:15.764180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:15.764180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:15.764180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.764180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:15.764180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:15.764180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:15.764180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.764180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:15.764180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:15.764180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:15.764180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:15.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:15.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:15.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:15.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:15.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:15.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:15.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:15.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:15.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:15.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:15.769510 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:15.769698 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:15.769825 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:15.769908 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:15.772468 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:15.772647 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:15.772852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.773200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:15.773200 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:15.773200 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:15.773200 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:15.773200 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:15.773200 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:15.773200 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.773200 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:15.773818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.773818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:15.773818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:15.773818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:15.773818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.773818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:15.773818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:15.773818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.773818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:15.773818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:15.773818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:15.773818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.774092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:15.774092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:15.774092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:15.774092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:15.774092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:15.774092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:15.774092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:15.774092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:15.774092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:15.774092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:15.774092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:15.774092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:15.774092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:15.774320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.774320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:15.774605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.774605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:15.774605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:15.774605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:15.774605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:15.774605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:15.774605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.774605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:15.774605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:15.774605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.774605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:15.774605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:15.774605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.774605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:15.774605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:15.774605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:15.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:15.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:15.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:15.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:15.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:15.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:15.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:15.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:15.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:15.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:15.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:15.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:15.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:15.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:15.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:15.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:15.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:15.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:15.776152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:15.776152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:15.776152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:18.663089 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:18.663737 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:18.663737 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:18.664194 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:18.714312 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:18.714526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:18.715524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:18.715524 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:18.715524 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:18.715524 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:18.715524 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:18.715524 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:18.715524 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:18.715524 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:18.716267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.052 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:18.716267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:18.716267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:18.716267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:18.716267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:18.716267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:18.716267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:18.716267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:18.716267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:18.716267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:18.716267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:18.716267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:18.716554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:18.716554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:18.716554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:18.716554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:18.716554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:18.716554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:18.716554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:18.716554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:18.716554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:18.716554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:18.716554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:18.716554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:18.716554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:18.716837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:18.716837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:18.717241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:18.717241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:18.717241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:18.717241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:18.717241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:18.717241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:18.717241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:18.717241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:18.717241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:18.717241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:18.717241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:18.717241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:18.717241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:18.717241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:18.717241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:18.717241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:18.717627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:18.717627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:18.717627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:18.717627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:18.717627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:18.717627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:18.717627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:18.717627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:18.717627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:18.717627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:18.717627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:18.717627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:18.717627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:18.717627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:18.717627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:18.717627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:18.717627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:18.717627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:18.717627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:18.718001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:18.718001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:18.718001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:20.733124 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:20.733124 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:20.733747 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:20.734603 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:20.734802 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:20.735073 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:20.735131 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:20.735223 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:20.735386 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:20.735884 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:20.735884 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:20.735884 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:20.735884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.736443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:20.736443 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:20.736443 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:20.736443 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:20.736443 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:20.736443 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:20.736443 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.736443 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:20.737550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.737550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:20.737550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:20.737550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:20.737550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.737550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:20.737550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:20.737550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.737550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:20.737550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:20.737550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:20.737550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.737960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:20.737960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:20.737960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:20.737960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:20.737960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:20.737960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:20.737960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:20.737960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:20.737960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:20.737960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:20.737960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:20.737960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:20.737960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:20.738394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.738394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:20.738974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.738974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:20.738974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:20.738974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:20.738974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:20.738974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:20.738974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.738974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:20.738974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:20.738974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.738974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:20.738974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:20.738974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.738974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:20.738974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:20.738974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.739920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:20.739920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:20.739920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.739920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:20.739920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:20.739920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:20.739920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.739920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:20.739920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:20.739920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:20.739920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:20.739920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:20.739920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:20.739920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:20.739920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:20.739920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:20.739920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:20.739920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:20.739920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:20.740451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:20.740451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.740451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:20.746498 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:20.746828 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:20.747121 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:20.747234 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:29:20.747858 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:20.747956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.751412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:20.751412 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:20.751412 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:20.751412 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:20.751412 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:20.751412 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:20.751412 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.751412 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:20.751412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.751412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:20.751412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:20.751412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:20.751412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.751412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:20.751412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:20.751412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:20.751412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.751412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:20.751986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:20.751986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:20.751986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:20.751986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:20.751986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:20.751986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:20.751986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:20.751986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:20.751986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:20.751986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:20.751986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:20.751986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:20.751986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.751986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:20.751986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.752484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:20.752484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:20.752484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:20.752484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.752484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:20.752484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:20.752484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:29:20.752484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:20.752484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:20.752484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.752484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:20.752484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:20.752484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:20.752905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.752905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:20.752905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:20.752905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:20.752905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:20.752905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:20.752905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:20.752905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:20.752905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:20.752905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:20.752905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:20.752905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:20.752905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:20.752905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:20.753362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.753362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:20.753362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.753362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:20.753362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:20.753362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:20.753362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:20.753362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:20.753362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.753362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:20.753362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:20.753362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.753362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:20.753362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:20.753362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.753818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:20.753818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:20.753818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.753818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:20.753818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:20.753818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:29:20.753818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:20.753818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:20.753818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.753818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:20.753818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:20.753818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:20.753818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.753818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:20.753818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:20.753818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:20.753818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:20.753818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:20.753818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:20.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:20.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:20.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:20.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:20.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:20.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:20.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:20.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:20.760500 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:20.760926 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:20.761109 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:20.761232 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:20.761742 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:20.761984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.762511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:20.762511 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:20.762511 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:20.762511 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:20.762511 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:20.762511 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:20.762511 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.762511 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:20.763548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.763548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:20.763548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:20.763548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:20.763548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.763548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:20.763548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:20.763548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:20.763548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.763548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:20.763548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:20.763548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:20.763548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:20.763548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:20.763548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:20.763790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:20.763790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:20.763790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:20.763790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:20.763790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:20.763790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:20.763790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:20.763790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.763790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:20.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:20.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:20.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:20.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:20.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:20.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:20.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:20.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:20.764668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:20.764668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:20.764668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.764668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:20.764668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:20.764668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:20.764668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:20.764668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:20.764668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:20.764668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:20.764668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:20.764668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:20.764668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:20.764668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:20.764915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:20.764915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:20.764915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.764915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:20.765249 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.765249 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:20.765249 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:20.765249 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:20.765249 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:20.765249 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:20.765249 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.765249 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:20.765249 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:20.765249 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.765249 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:20.765249 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:20.765249 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.765249 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:20.765249 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:20.765249 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.765249 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:20.765580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:20.765580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:20.765580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:20.765580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.765580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:20.765580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:20.765580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:20.765580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.765580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:20.765580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:20.765580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:20.765580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:20.765580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:20.765580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:20.765580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:20.765580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:20.765580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:20.765580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:20.765580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:20.765915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:20.765915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:20.765915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.765915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:20.769850 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:20.770273 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:20.770417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:20.770582 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:20.771000 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:20.771277 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:20.771516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.772007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:20.772007 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:20.772007 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:20.772007 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:20.772007 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:20.772007 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:20.772007 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.772007 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:20.772719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.772719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:20.772719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:20.772719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:20.772719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.772719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:20.772719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:20.772719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.772719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:20.772719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:20.772719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:20.772719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.772962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:20.772962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:20.772962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:20.772962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:20.772962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:20.772962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:20.772962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:20.772962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:20.772962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:20.772962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:20.772962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:20.772962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:20.772962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:20.773197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.773197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:20.773484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.773484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:20.773484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:20.773484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:20.773484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:20.773484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:20.773484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.773484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:20.773484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:20.773484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.773484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:20.773484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:20.773484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.773484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:20.773484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:20.773484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:20.773760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:20.773760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:20.773760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.773760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:20.773760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:20.773760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:20.773760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:20.773760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:20.773760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:20.773760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:20.773760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:20.773760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:20.773760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:20.773760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:20.773760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:20.773760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:20.773760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:20.773760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:20.773760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:20.774129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:20.774129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:20.774129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:23.663650 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:23.664121 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:23.664121 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:23.664647 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:23.702571 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:23.702767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:23.703569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:23.703569 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:23.703569 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:23.703569 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:23.703569 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:23.703569 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:23.703569 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:23.703569 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:23.704455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:23.704455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:23.704455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:23.704455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:23.704455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:23.704455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:23.704455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:23.704455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:23.704455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:23.704455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:23.704455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:23.704455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:23.704707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:23.704707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:23.704707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:23.704707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:23.704707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:23.704707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:23.704707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:23.704707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:23.704707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:23.704707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:23.704707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:23.704707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:23.704707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:23.704941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:23.704941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:23.705413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:23.705413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:23.705413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:23.705413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:23.705413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:23.705413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:23.705413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:23.705413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:23.705413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:23.705413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:23.705413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:23.705413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:23.705413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:23.705413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:23.705413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:23.705413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:23.705727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:23.705727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:23.705727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:23.705727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:23.705727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:23.705727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:23.705727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:23.705727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:23.705727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:23.705727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:23.705727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:23.705727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:23.705727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:23.705727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:23.705727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:23.705727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:23.705727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:23.705727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:23.705727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:23.706083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:23.706083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:23.706083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:25.735721 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:25.736280 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:25.737440 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:25.737591 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:25.738966 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:25.738966 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:25.738966 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:25.738966 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:25.738966 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:25.739237 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:25.739237 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:25.739237 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:25.739237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.739939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:25.739939 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:25.739939 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:25.739939 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:25.739939 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:25.739939 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:25.739939 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.739939 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:25.741485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.741485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:25.741485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:25.741485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:25.741485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.741485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:25.741485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:25.741485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.741485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:25.741485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:25.741485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:25.741485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.741924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:25.741924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:25.741924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:25.741924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:25.741924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:25.741924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:25.741924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:25.741924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:25.741924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:25.741924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:25.741924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:25.741924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:25.741924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:25.742365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.742365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:25.747279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.747279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:25.747279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:25.747279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:25.747279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:25.747279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:25.747279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.747279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:25.747279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:25.747279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.747279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:25.747279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:25.747279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.747279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:25.747279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:25.747279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.747900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:25.747900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:25.747900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.747900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:25.747900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:25.747900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:25.747900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.747900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:25.747900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:25.747900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:25.747900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:25.747900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:25.747900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:25.747900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:25.747900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:25.747900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:25.747900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:25.747900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:25.747900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:25.748520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:25.748520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.748520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:25.748520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:25.748520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:25.748520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:25.748520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:29:25.748520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:25.748787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.749100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:25.749100 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:25.749100 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:25.749100 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:25.749100 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:25.749100 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:25.749100 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.749100 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:25.751382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.751382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:25.751382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:25.751382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:25.751382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.751382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:25.751382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:25.751382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:25.751382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.751382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:25.751382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:25.751382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:25.751382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:25.751382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:25.751382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:25.753145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:25.753145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:25.753145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:25.753145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:25.753145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:25.753145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:25.753145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:25.753145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.753145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:25.755992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.755992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:25.755992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:25.755992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:25.755992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.755992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:25.755992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:25.755992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:29:25.755992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:25.755992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:25.755992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.755992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:25.756285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:25.756285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:25.756285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.756285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:25.756285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:25.756285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:25.756285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:25.756285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:25.756285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:25.756285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:25.756285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:25.756285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:25.756285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:25.756285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:25.756558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:25.756558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:25.756558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.756558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:25.759970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.759970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:25.759970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:25.759970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:25.759970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:25.759970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:25.759970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.759970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:25.759970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:25.759970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.759970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:25.759970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:25.759970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.759970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:25.759970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:25.759970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.760377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:25.760377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:25.760377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:29:25.760377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:25.760377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:25.760377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.760377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:25.760377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:25.760377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:25.760377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.760377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:25.760377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:25.760377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:25.760377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:25.760377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:25.760377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:25.760377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:25.760377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:25.760702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:25.760702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:25.760702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:25.760702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:25.760702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:25.760702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.760702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:25.764275 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:25.764275 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:25.764275 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:25.764275 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:25.764275 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:25.764275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.764489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:25.764489 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:25.764489 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:25.764489 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:25.764489 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:25.764489 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:25.764489 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.764489 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:25.764489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.764489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:25.764489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:25.764489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:25.764489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.764489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:25.764489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:25.764489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:25.764489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.764489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:25.764830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:25.764830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:25.764830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:25.764830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:25.764830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:25.764830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:25.764830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:25.764830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:25.764830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:25.764830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:25.764830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:25.764830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:25.764830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.764830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:25.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:25.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:25.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:25.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:25.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:25.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:25.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:25.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.765355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:25.765653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:25.765653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:25.765653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.765653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:25.765653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:25.765653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:25.765653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:25.765653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:25.765653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:25.765653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:25.765653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:25.765653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:25.765653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:25.765653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:25.765896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:25.765896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:25.765896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.765896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:25.767849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.767849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:25.767849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:25.767849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:25.767849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:25.767849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:25.767849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.767849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:25.767849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:25.767849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.767849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:25.767849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:25.767849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.767849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:25.767849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:25.767849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.767849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:25.768210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:25.768210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:25.768210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:25.768210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.768210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:25.768210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:25.768210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:25.768210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.768210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:25.768210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:25.768210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:25.768210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:25.768210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:25.768210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:25.768210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:25.768210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:25.768210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:25.768210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:25.768210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:25.768549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:25.768549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:25.768549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.768549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:25.770494 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:25.770786 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:25.770931 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:25.771131 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:25.771438 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:25.771629 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:25.771855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.772324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:25.772324 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:25.772324 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:25.772324 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:25.772324 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:25.772324 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:25.772324 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.772324 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:25.774438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.774438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:25.774438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:25.774438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:25.774438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.774438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:25.774438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:25.774438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.774438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:25.774438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:25.774438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:25.774438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.774765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:25.774765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:25.774765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:25.774765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:25.774765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:25.774765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:25.774765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:25.774765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:25.774765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:25.774765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:25.774765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:25.774765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:25.774765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:25.775033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.775033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:25.776476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.776476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:25.776476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:25.776476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:25.776476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:25.776476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:25.776476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.776476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:25.776476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:25.776476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.776476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:25.776476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:25.776476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.776476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:25.776476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:25.776476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:25.776861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:25.776861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:25.776861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.776861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:25.776861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:25.776861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:25.776861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:25.776861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:25.776861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:25.776861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:25.776861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:25.776861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:25.776861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:25.776861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:25.776861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:25.776861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:25.776861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:25.776861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:25.776861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:25.777215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:25.777215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:25.777215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:28.664199 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:28.665296 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:28.665296 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:28.665296 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:28.709885 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:28.710718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:28.710972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:28.710972 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:28.710972 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:28.710972 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:28.710972 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:28.710972 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:28.710972 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:28.710972 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:28.712642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:28.712642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:28.712642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:28.712642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:28.712642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:28.712642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:28.712642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:28.712642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:28.712642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:28.712642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:28.712642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:28.712642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:28.713004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:28.713004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:28.713004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:28.713004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:28.713004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:28.713004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:28.713004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:28.713004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:28.713004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:28.713004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:28.713004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:28.713004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:28.713004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:28.713443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:28.713443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:28.713443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:28.713443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:28.713443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:28.713443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:28.713443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:28.713443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:28.713443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:28.713443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:28.713443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:28.713443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:28.713443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:28.713443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:28.713443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:28.713927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:28.713927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:28.713927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:28.713927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:28.713927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:28.713927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:28.713927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:28.713927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:28.713927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:28.713927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:28.713927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:28.713927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:28.713927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:28.713927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:28.713927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:28.713927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:28.713927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:28.713927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:28.713927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:28.714505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:28.714505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:28.714505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:28.714505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:28.714505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:28.714505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:30.736443 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:30.736935 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:30.736935 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:30.738176 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:30.738408 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:30.738557 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:30.738692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:30.738827 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:30.738954 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:30.739092 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:30.739265 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:30.739360 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:30.739485 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:30.740195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.741530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:30.741530 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:30.741530 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:30.741530 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:30.741530 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:30.741530 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:30.741530 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.741530 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:30.742498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.742498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:30.742498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:30.742498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:30.742498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.742498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:30.742498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:30.742498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.742498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:30.742498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:30.742498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:30.742498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.742854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:30.742854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:30.742854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:30.742854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:30.742854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:30.742854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:30.742854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:30.742854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:30.742854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:30.742854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:30.742854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:30.742854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:30.742854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:30.743204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.743204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:30.743280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.743280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:30.743280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:30.743280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:30.743280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:30.743280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:30.743280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.743280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:30.743280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:30.743280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.743280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:30.743280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:30.743280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.743280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:30.743280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:30.743280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.743685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:30.743685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:30.743685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.743685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:30.743685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:30.743685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:30.743685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.743685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:30.743685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:30.743685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:30.743685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:30.743685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:30.743685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:30.743685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:30.743685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:30.743685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:30.743685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:30.743685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:30.743685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:30.744377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:30.744377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.744377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:30.748210 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:30.748210 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:30.748210 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:30.748210 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:29:30.748210 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:30.748210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.748574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:30.748574 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:30.748574 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:30.748574 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:30.748574 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:30.748574 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:30.748574 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.748574 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:30.749103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.749103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:30.749103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:30.749103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:30.749103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.749103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:30.749103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:30.749103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:30.749103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.749103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:30.749103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:30.749103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:30.749103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:30.749103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:30.749103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:30.749580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:30.749580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:30.749580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:30.749580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:30.749580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:30.749580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:30.749580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:30.749580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.749580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:30.749822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.749822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:30.749822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:30.749822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:30.749822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.749822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:30.749822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:30.749822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:29:30.749822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:30.749822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:30.749822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.749822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:30.750157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:30.750157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:30.750157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.750157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:30.750157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:30.750157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:30.750157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:30.750157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:30.750157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:30.750157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:30.750157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:30.750157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:30.750157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:30.750157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:30.750524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:30.750524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:30.750524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.750524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:30.750524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.750524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:30.750524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:30.750524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:30.750524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:30.750524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:30.750524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.750524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:30.750524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:30.750524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.750886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:30.750886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:30.750886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.750886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:30.750886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:30.750886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.750886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:30.750886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:30.750886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:29:30.750886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:30.750886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:30.750886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.750886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:30.750886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:30.750886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:30.750886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.750886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:30.750886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:30.750886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:30.751423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:30.751423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:30.751423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:30.751423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:30.751423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:30.751423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:30.751423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:30.751423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:30.751423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:30.751423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:30.751423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.751423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:30.762064 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:30.762064 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:30.762064 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:30.762064 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:30.762311 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:30.762507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.763004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:30.763004 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:30.763004 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:30.763004 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:30.763004 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:30.763004 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:30.763004 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.763004 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:30.763744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.763744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:30.763744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:30.763744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:30.763744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.763744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:30.763744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:30.763744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:30.763744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.763744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:30.763744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:30.763744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:30.763744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:30.763744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:30.763744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:30.764050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:30.764050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:30.764050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:30.764050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:30.764050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:30.764050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:30.764050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:30.764050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.764050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:30.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:30.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:30.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:30.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:30.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:30.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:30.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:30.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.764461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:30.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:30.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:30.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:30.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:30.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:30.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:30.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:30.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:30.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:30.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:30.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:30.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:30.764670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:30.765226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:30.765226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:30.765226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.765226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:30.765226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.765226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:30.765226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:30.765226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:30.765226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:30.765226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:30.765226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.765226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:30.765226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:30.765226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.765535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:30.765535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:30.765535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.765535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:30.765535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:30.765535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.765535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:30.765535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:30.765535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:30.765535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:30.765535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.765535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:30.765535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:30.765535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:30.765535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.765535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:30.765535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:30.765535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:30.765535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:30.765843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:30.765843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:30.765843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:30.765843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:30.765843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:30.765843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:30.765843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:30.765843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:30.765843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:30.765843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.765843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:30.770946 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:30.771348 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:30.771536 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:30.771785 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:30.774345 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:30.775059 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:30.776245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.777737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:30.777737 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:30.777737 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:30.777737 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:30.777737 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:30.777737 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:30.777737 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.777737 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:30.778735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.778735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:30.778735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:30.778735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:30.778735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.778735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:30.778735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:30.778735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.778735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:30.778735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:30.778735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:30.778735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.779200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:30.779200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:30.779200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:30.779200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:30.779200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:30.779200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:30.779200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:30.779200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:30.779200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:30.779200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:30.779200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:30.779200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:30.779200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:30.779625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.779625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:30.780456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.780456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:30.780456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:30.780456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:30.780456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:30.780456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:30.780456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.780456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:30.780456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:30.780456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.780456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:30.780456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:30.780456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.780456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:30.780456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:30.780456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:30.781028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:30.781028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:30.781028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.781028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:30.781028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:30.781028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:30.781028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:30.781028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:30.781028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:30.781028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:30.781028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:30.781028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:30.781028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:30.781028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:30.781028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:30.781028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:30.781028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:30.781028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:30.781028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:30.781594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:30.781594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:30.781594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:33.664570 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:33.665022 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:33.665022 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:33.666308 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:33.713844 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:33.714037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:33.714480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:33.714480 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:33.714480 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:33.714480 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:33.714480 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:33.714480 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:33.714480 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:33.714480 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:33.715117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.049 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:33.715117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:33.715117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:33.715117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:33.715117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:33.715117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:33.715117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:33.715117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:33.715117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:33.715117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:33.715117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:33.715117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:33.715389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:33.715389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:33.715389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:33.715389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:33.715389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:33.715389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:33.715389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:33.715389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:33.715389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:33.715389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:33.715389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:33.715389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:33.715389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:33.715633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:33.715633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:33.715833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:33.715833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:33.715833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:33.715833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:33.715833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:33.715833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:33.715833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:33.715833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:33.715833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:33.715833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:33.715833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:33.715833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:33.715833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:33.715833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:33.715833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:33.715833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:33.716106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:33.716106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:33.716106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:33.716106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:33.716106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:33.716106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:33.716106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:33.716106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:33.716106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:33.716106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:33.716106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:33.716106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:33.716106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:33.716106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:33.716106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:33.716106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:33.716106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:33.716106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:33.716106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:33.716894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:33.716894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:33.716894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:35.736858 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:35.737388 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:35.737388 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:35.738311 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:35.738508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c1f19fc-7fd3-48c7-93c2-4a8639cf9658, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:35.738685 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:35.738852 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:35.739009 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:35.739168 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:35.739335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:35.739490 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b2843c0-8cbc-4047-beee-b02a21f2c90f, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:35.739638 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:35.739782 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:35.740109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.741454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:35.741454 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:35.741454 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:35.741454 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:35.741454 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:35.741454 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:35.741454 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.741454 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:35.741454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.741454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:35.741454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:35.741454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:35.741454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.741454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:35.741454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:35.741771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.741771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:35.741771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:35.741771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:35.741771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.741771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:35.741771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:35.741771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:35.741771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:35.741771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:35.741771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:35.741771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:35.741771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:35.741771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:35.742003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:35.742003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:35.742003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:35.742003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:35.742003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.742003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:35.742554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.742554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:35.742554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:35.742554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:35.742554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:35.742554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:35.742554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.742554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:35.742554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:35.742554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.742554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:35.742554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:35.742554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.742554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:35.742554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:35.742554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.742852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:35.742852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:35.742852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.742852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:35.742852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:35.742852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:35.742852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.742852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:35.742852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:35.742852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:35.742852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:35.742852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:35.742852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:35.742852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:35.742852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:35.742852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:35.742852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:35.742852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:35.742852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:35.743187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:35.743187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.743187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:35.747482 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:35.747818 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:35.747818 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:35.748007 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:29:35.748319 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:35.748521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.749059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:35.749059 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:35.749059 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:35.749059 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:35.749059 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:35.749059 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:35.749059 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.749059 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:35.749600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.749600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:35.749600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:35.749600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:35.749600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.749600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:35.749600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:35.749600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:35.749600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.749600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:35.749600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:35.749600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:35.749600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:35.749600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:35.749600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:35.749868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:35.749868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:35.749868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:35.749868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:35.749868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:35.749868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:35.749868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:35.749868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.749868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:35.750748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.750748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:35.750748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:35.750748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:35.750748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.750748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:35.750748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:35.750748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:29:35.750748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:35.750748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:35.750748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.750748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:35.750996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:35.750996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:35.750996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.750996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:35.750996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:35.750996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:35.750996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:35.750996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:35.750996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:35.750996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:35.750996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:35.750996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:35.750996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:35.750996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:35.751238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:35.751238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:35.751238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.751238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:35.752130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.752130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:35.752130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:35.752130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:35.752130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:35.752130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:35.752130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.752130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:35.752130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:35.752130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.752130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:35.752130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:35.752130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.752130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:35.752130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:35.752130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.752551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:35.752551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:35.752551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:29:35.752551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:35.752551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:35.752551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.752551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:35.752551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:35.752551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:35.752551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.752551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:35.752551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:35.752551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:35.752551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:35.752551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:35.752551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:35.752551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:35.752551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:35.752868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:35.752868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:35.752868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:35.752868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:35.752868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:35.752868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.752868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:35.762081 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:35.762148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:35.762216 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:35.762331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:35.762677 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:35.762915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.763328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:35.763328 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:35.763328 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:35.763328 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:35.763328 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:35.763328 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:35.763328 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.763328 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:35.763961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.763961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:35.763961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:35.763961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:35.763961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.763961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:35.763961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:35.763961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:35.763961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.763961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:35.763961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:35.763961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:35.763961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:35.763961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:35.763961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:35.764299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:35.764299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:35.764299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:35.764299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:35.764299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:35.764299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:35.764299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:35.764299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.764299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:35.765114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.765114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:35.765114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:35.765114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:35.765114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.765114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:35.765114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:35.765114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:35.765114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:35.765114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.765114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:35.765352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:35.765352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:35.765352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.765352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:35.765352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:35.765352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:35.765352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:35.765352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:35.765352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:35.765352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:35.765352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:35.765352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:35.765352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:35.765352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:35.765590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:35.765590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:35.765590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.765590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:35.766157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.766157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:35.766157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:35.766157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:35.766157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:35.766157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:35.766157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.766157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:35.766157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:35.766157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.766157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:35.766157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:35.766157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.766157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:35.766157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:35.766157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.766157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:35.766462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:35.766462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:35.766462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:35.766462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.766462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:35.766462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:35.766462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:35.766462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.766462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:35.766462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:35.766462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:35.766462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:35.766462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:35.766462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:35.766462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:35.766462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:35.766462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:35.766462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:35.766462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:35.766766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:35.766766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:35.766766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.766766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:35.771377 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:35.771587 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:35.771685 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:35.771807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:35.772136 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:35.772577 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:35.772766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.773096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:35.773096 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:35.773096 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:35.773096 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:35.773096 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:35.773096 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:35.773096 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.773096 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:35.773796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.773796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:35.773796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:35.773796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:35.773796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.773796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:35.773796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:35.773796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.773796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:35.773796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:35.773796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:35.773796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.774040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:35.774040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:35.774040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:35.774040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:35.774040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:35.774040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:35.774040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:35.774040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:35.774040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:35.774040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:35.774040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:35.774040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:35.774040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:35.774302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.774302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:35.775105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.775105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:35.775105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:35.775105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:35.775105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:35.775105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:35.775105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.775105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:35.775105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:35.775105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.775105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:35.775105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:35.775105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.775105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:35.775105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:35.775105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:35.775512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:35.775512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:35.775512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.775512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:35.775512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:35.775512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:35.775512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:35.775512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:35.775512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:35.775512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:35.775512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:35.775512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:35.775512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:35.775512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:35.775512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:35.775512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:35.775512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:35.775512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:35.775512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:35.775837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:35.775837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:35.775837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:38.665550 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:38.665550 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:38.665550 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:38.666295 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:38.706090 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:38.706357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:38.706825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:38.706825 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:38.706825 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:38.706825 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:38.706825 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:38.706825 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:38.706825 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:38.706825 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:38.707679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:38.707679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:38.707679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:38.707679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:38.707679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:38.707679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:38.707679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:38.707679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:38.707679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:38.707679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:38.707679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:38.707679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:38.708505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:38.708505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:38.708505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:38.708505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:38.708505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:38.708505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:38.708505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:38.708505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:38.708505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:38.708505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:38.708505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:38.708505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:38.708505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:38.709286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:38.709286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:38.709286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:38.709286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:38.709286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:38.709286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:38.709286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:38.709286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:38.709286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:38.709286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:38.709286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:38.709286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:38.709286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:38.709286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:38.709286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:38.712388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:38.712388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:38.712388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:38.712388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:38.712388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:38.712388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:38.712388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:38.712388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:38.712388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:38.712388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:38.712388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:38.712388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:38.712388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:38.712388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:38.712388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:38.712388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:38.712388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:38.712388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:38.712388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:38.712935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:38.712935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:38.712935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:38.712935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:38.712935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:38.712935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:40.737367 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:40.737884 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:40.737884 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:40.738501 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:40.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:40.738781 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:40.738895 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:40.739064 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:40.739150 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:40.739297 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:40.739437 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:40.739558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.740028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:40.740028 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:40.740028 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:40.740028 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:40.740028 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:40.740028 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:40.740028 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.740028 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:40.741354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.741354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:40.741354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:40.741354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:40.741354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.741354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:40.741354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:40.741354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.741354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:40.741354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:40.741354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:40.741354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.741752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:40.741752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:40.741752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:40.741752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:40.741752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:40.741752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:40.741752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:40.741752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:40.741752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:40.741752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:40.741752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:40.741752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:40.741752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:40.742120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.742120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:40.742120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.742120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:40.742120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:40.742120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:40.742120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:40.742120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:40.742120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.742120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:40.742120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:40.742120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.742120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:40.742120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:40.742120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.742592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:40.742592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:40.742592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.742592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:40.742592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:40.742592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.742592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:40.742592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:40.742592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:40.742592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.742592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:40.742592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:40.742592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:40.742592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:40.742592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:40.742592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:40.742592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:40.742592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:40.742592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:40.743112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:40.743112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:40.743112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:40.743112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:40.743112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.743112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:40.747876 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:40.749469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:40.749552 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:40.749638 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:29:40.750176 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:40.750339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.751450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:40.751450 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:40.751450 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:40.751450 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:40.751450 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:40.751450 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:40.751450 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.751450 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:40.752029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.752029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:40.752029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:40.752029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:40.752029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.752029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:40.752029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:40.752029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:40.752029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.752029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:40.752029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:40.752029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:40.752029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:40.752029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:40.752029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:40.752627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:40.752627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:40.752627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:40.752627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:40.752627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:40.752627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:40.752627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:40.752627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.752627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:40.752939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.752939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:40.752939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:40.752939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:40.752939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.752939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:40.752939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:40.752939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:29:40.752939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:40.752939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:40.752939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.752939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:40.753392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:40.753392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:40.753392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.753392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:40.753392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:40.753392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:40.753392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:40.753392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:40.753392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:40.753392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:40.753392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:40.753392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:40.753392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:40.753392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:40.753812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:40.753812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:40.753812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.753812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:40.754345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.754345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:40.754345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:40.754345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:40.754345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:40.754345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:40.754345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.754345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:40.754345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:40.754345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.754345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:40.754345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:40.754345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.754345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:40.754345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:40.754345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.754828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:40.754828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:40.754828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:29:40.754828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:40.754828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:40.754828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.754828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:40.754828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:40.754828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:40.754828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.754828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:40.754828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:40.754828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:40.754828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:40.754828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:40.754828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:40.754828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:40.754828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:40.755439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:40.755439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:40.755439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:40.755439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:40.755439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:40.755439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.755439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:40.762375 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:40.762661 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:40.762762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:40.762857 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:40.763180 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:40.763389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.763772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:40.763772 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:40.763772 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:40.763772 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:40.763772 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:40.763772 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:40.763772 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.763772 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:40.764347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.764347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:40.764347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:40.764347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:40.764347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.764347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:40.764347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:40.764347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:40.764347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.764347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:40.764347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:40.764347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:40.764347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:40.764347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:40.764347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:40.764617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:40.764617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:40.764617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:40.764617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:40.764617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:40.764617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:40.764617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:40.764617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.764617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:40.765548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.765548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:40.765548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:40.765548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:40.765548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.765548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:40.765548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:40.765548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:40.765548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:40.765548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.765548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:40.765787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:40.765787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:40.765787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.765787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:40.765787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:40.765787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:40.765787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:40.765787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:40.765787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:40.765787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:40.765787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:40.765787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:40.765787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:40.765787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:40.766077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:40.766077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:40.766077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.766077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:40.766332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.766332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:40.766332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:40.766332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:40.766332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:40.766332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:40.766332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.766332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:40.766332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:40.766332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.766332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:40.766332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:40.766332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.766332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:40.766332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:40.766332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.766332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:40.766640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:40.766640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:40.766640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:40.766640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.766640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:40.766640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:40.766640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:40.766640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.766640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:40.766640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:40.766640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:40.766640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:40.766640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:40.766640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:40.766640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:40.766640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:40.766640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:40.766640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:40.766640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:40.766971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:40.766971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:40.766971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.766971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:40.772262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:40.772605 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:40.773476 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:40.773476 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:40.777339 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:40.777339 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:40.777339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.777457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:40.777457 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:40.777457 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:40.777457 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:40.777457 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:40.777457 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:40.777457 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.777457 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:40.778169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.778169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:40.778169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:40.778169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:40.778169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.778169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:40.778169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:40.778169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.778169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:40.778169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:40.778169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:40.778169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.778446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:40.778446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:40.778446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:40.778446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:40.778446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:40.778446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:40.778446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:40.778446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:40.778446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:40.778446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:40.778446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:40.778446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:40.778446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:40.778677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.778677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:40.778999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.778999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:40.778999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:40.778999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:40.778999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:40.778999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:40.778999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.778999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:40.778999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:40.778999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.778999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:40.778999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:40.778999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.778999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:40.778999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:40.778999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:40.779329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:40.779329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:40.779329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.779329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:40.779329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:40.779329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:40.779329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:40.779329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:40.779329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:40.779329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:40.779329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:40.779329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:40.779329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:40.779329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:40.779329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:40.779329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:40.779329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:40.779329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:40.779329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:40.779628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:40.779628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:40.779628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:43.668654 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:43.669202 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:43.669202 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:43.671814 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:43.715796 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:43.715960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:43.716667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:43.716667 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:43.716667 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:43.716667 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:43.716667 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:43.716667 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:43.716667 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:43.716667 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:43.717405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:43.717405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:43.717405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:43.717405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:43.717405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:43.717405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:43.717405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:43.717405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:43.717405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:43.717405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:43.717405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:43.717405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:43.717955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:43.717955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:43.717955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:43.717955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:43.717955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:43.717955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:43.717955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:43.717955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:43.717955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:43.717955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:43.717955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:43.717955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:43.717955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:43.718658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:43.718658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:43.718658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:43.718658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:43.718658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:43.718658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:43.718658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:43.718658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:43.718658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:43.718658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:43.718658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:43.718658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:43.718658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:43.718658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:43.718658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:43.719352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:43.719352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:43.719352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:43.719352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:43.719352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:43.719352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:43.719352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:43.719352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:43.719352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:43.719352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:43.719352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:43.719352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:43.719352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:43.719352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:43.719352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:43.719352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:43.719352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:43.719352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:43.719352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:43.720004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:43.720004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:43.720004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:43.720004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:43.720004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:43.720004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:45.737766 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:45.739674 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:45.739674 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:45.739934 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:45.740106 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:45.740217 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:45.740369 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:45.740465 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:45.740565 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:45.740660 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:45.740756 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:45.740859 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:45.740966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.741600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:45.741600 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:45.741600 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:45.741600 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:45.741600 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:45.741600 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:45.741600 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.741600 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:45.744097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.744097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:45.744097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:45.744097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:45.744097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.744097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:45.744097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:45.744097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.744097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:45.744097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:45.744097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:45.744097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.744510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:45.744510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:45.744510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:45.744510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:45.744510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:45.744510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:45.744510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:45.744510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:45.744510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:45.744510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:45.744510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:45.744510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:45.744510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:45.745128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.745128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:45.747467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.747467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:45.747467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:45.747467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:45.747467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:45.747467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:45.747467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.747467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:45.747467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:45.747467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.747467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:45.747467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:45.747467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.747467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:45.747467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:45.747467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.747823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:45.747823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:45.747823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.747823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:45.747823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:45.747823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:45.747823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.747823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:45.747823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:45.747823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:45.747823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:45.747823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:45.747823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:45.747823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:45.747823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:45.747823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:45.747823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:45.747823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:45.747823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:45.748272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:45.748272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.748272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:45.749494 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:45.749827 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:45.749990 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:45.750845 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:29:45.751171 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:45.751350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.751799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:45.751799 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:45.751799 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:45.751799 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:45.751799 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:45.751799 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:45.751799 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.751799 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:45.753410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.753410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:45.753410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:45.753410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:45.753410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.753410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:45.753410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:45.753410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:45.753410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.753410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:45.753410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:45.753410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:45.753410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:45.753410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:45.753410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:45.753768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:45.753768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:45.753768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:45.753768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:45.753768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:45.753768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:45.753768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:45.753768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.753768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:45.755299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.755299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:45.755299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:45.755299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:45.755299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.755299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:45.755299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:45.755299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:29:45.755299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:45.755299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:45.755299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.755299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:45.755611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:45.755611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:45.755611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.755611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:45.755611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:45.755611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:45.755611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:45.755611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:45.755611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:45.755611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:45.755611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:45.755611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:45.755611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:45.755611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:45.755981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:45.755981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:45.755981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.755981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:45.755981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.755981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:45.755981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:45.755981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:45.755981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:45.755981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:45.755981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.755981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:45.755981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:45.755981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.756313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:45.756313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:45.756313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.756313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:45.756313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:45.756313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.756313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:45.756313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:45.756313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:29:45.756313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:45.756313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:45.756313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.756313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:45.756313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:45.756313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:45.756313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.756313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:45.756313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:45.756313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:45.756654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:45.756654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:45.756654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:45.756654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:45.756654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:45.756654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:45.756654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:45.756654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:45.756654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:45.756654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:45.756654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.756654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:45.762736 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:45.762969 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:45.763107 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:45.763173 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:45.764212 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:45.764709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.765641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:45.765641 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:45.765641 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:45.765641 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:45.765641 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:45.765641 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:45.765641 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.765641 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:45.766488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.766488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:45.766488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:45.766488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:45.766488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.766488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:45.766488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:45.766488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:45.766488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.766488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:45.766488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:45.766488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:45.766488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:45.766488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:45.766488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:45.766784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:45.766784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:45.766784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:45.766784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:45.766784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:45.766784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:45.766784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:45.766784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.766784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:45.767442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.767442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:45.767442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:45.767442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:45.767442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.767442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:45.767442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:45.767442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:45.767442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:45.767442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.767442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:45.767679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:45.767679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:45.767679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.767679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:45.767679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:45.767679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:45.767679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:45.767679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:45.767679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:45.767679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:45.767679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:45.767679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:45.767679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:45.767679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:45.767909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:45.767909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:45.767909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.767909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:45.768944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.768944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:45.768944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:45.768944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:45.768944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:45.768944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:45.768944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.768944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:45.768944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:45.768944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.768944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:45.768944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:45.768944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.768944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:45.768944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:45.768944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.768944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:45.769401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:45.769401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:45.769401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:45.769401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.769401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:45.769401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:45.769401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:45.769401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.769401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:45.769401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:45.769401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:45.769401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:45.769401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:45.769401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:45.769401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:45.769401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:45.769401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:45.769401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:45.769401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:45.769798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:45.769798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:45.769798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.769798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:45.772652 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:45.772848 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:45.772952 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:45.773100 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:45.774772 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:45.775095 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:45.775423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.775938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:45.775938 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:45.775938 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:45.775938 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:45.775938 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:45.775938 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:45.775938 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.775938 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:45.776535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.776535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:45.776535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:45.776535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:45.776535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.776535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:45.776535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:45.776535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.776535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:45.776535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:45.776535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:45.776535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.776805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:45.776805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:45.776805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:45.776805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:45.776805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:45.776805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:45.776805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:45.776805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:45.776805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:45.776805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:45.776805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:45.776805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:45.776805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:45.777059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.777059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:45.778196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.778196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:45.778196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:45.778196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:45.778196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:45.778196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:45.778196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.778196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:45.778196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:45.778196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.778196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:45.778196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:45.778196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.778196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:45.778196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:45.778196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:45.778498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:45.778498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:45.778498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.778498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:45.778498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:45.778498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:45.778498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:45.778498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:45.778498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:45.778498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:45.778498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:45.778498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:45.778498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:45.778498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:45.778498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:45.778498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:45.778498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:45.778498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:45.778498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:45.778864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:45.778864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:45.778864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:48.668868 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:48.669517 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:48.669517 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:48.669738 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:48.733120 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:48.733699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:48.734280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:48.734280 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:48.734280 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:48.734280 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:48.734280 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:48.734280 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:48.734280 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:48.734280 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:48.736753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.065 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:48.736753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:48.736753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:48.736753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:48.736753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:48.736753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:48.736753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:48.736753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:48.736753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:48.736753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:48.736753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:48.736753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:48.737106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:48.737106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:48.737106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:48.737106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:48.737106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:48.737106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:48.737106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:48.737106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:48.737106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:48.737106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:48.737106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:48.737106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:48.737106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:48.737578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:48.737578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:48.738259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:48.738259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:48.738259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:48.738259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:48.738259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:48.738259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:48.738259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:48.738259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:48.738259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:48.738259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:48.738259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:48.738259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:48.738259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:48.738259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:48.738259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:48.738259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:48.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:48.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:48.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:48.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:48.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:48.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:48.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:48.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:48.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:48.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:48.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:48.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:48.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:48.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:48.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:48.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:48.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:48.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:48.738652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:48.739075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:48.739075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:48.739075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:50.738997 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:50.739510 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:50.739630 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:50.740821 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:50.741028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:50.741189 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:50.741335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:50.742971 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:50.742971 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:50.742971 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:50.743077 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:50.743171 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:50.743640 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:50.743640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.743900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:50.743900 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:50.743900 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:50.743900 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:50.743900 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:50.743900 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:50.743900 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.743900 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:50.744947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.744947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:50.744947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:50.744947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:50.744947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.744947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:50.744947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:50.744947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.744947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:50.744947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:50.744947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:50.744947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.745268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:50.745268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:50.745268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:50.745268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:50.745268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:50.745268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:50.745268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:50.745268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:50.745268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:50.745268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:50.745268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:50.745268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:50.745268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:50.745495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.745495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:50.746293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.746293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:50.746293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:50.746293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:50.746293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:50.746293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:50.746293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.746293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:50.746293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:50.746293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.746293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:50.746293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:50.746293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.746293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:50.746293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:50.746293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.746713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:50.746713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:50.746713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.746713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:50.746713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:50.746713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:50.746713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.746713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:50.746713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:50.746713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:50.746713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:50.746713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:50.746713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:50.746713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:50.746713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:50.746713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:50.746713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:50.746713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:50.746713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:50.747057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:50.747057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.747057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:50.750891 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:50.750891 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:50.750891 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:50.750891 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:29:50.755692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:50.755907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.756382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:50.756382 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:50.756382 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:50.756382 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:50.756382 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:50.756382 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:50.756382 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.756382 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:50.756963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.756963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:50.756963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:50.756963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:50.756963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.756963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:50.756963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:50.756963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:50.756963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.756963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:50.756963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:50.756963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:50.756963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:50.756963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:50.756963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:50.757304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:50.757304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:50.757304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:50.757304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:50.757304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:50.757304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:50.757304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:50.757304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.757304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:50.757686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.757686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:50.757686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:50.757686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:50.757686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.757686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:50.757686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:50.757686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:29:50.757686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:50.757686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:50.757686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.757686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:50.760629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:50.760629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:50.760629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.760629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:50.760629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:50.760629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:50.760629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:50.760629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:50.760629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:50.760629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:50.760629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:50.760629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:50.760629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:50.760629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:50.760903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:50.760903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:50.760903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.760903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:50.760903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.760903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:50.760903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:50.760903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:50.760903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:50.760903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:50.760903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.760903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:50.760903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:50.760903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.761181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:50.761181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:50.761181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.761181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:50.761181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:50.761181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.761181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:50.761181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:50.761181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:29:50.761181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:50.761181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:50.761181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.761181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:50.761181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:50.761181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:50.761181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.761181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:50.761181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:50.761181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:50.761605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:50.761605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:50.761605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:50.761605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:50.761605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:50.761605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:50.761605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:50.761605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:50.761605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:50.761605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:50.761605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.761605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:50.765321 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:50.765645 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:50.765735 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:50.765865 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:50.773097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:50.773903 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:50.775304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.775455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:50.775455 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:50.775455 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:50.775455 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:50.775455 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:50.775455 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:50.775455 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.775455 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:50.776118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.776118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:50.776118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:50.776118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:50.776118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.776118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:50.776118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:50.776118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:50.776118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.776118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:50.776118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:50.776118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:50.776118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:50.776118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:50.776118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:50.776766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:50.776766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:50.776766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:50.776766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:50.776766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:50.776766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:50.776766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:50.776766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.776766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:50.776766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.776766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:50.776766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:50.776766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:50.776766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.777241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:50.777241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:50.777241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:50.777241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:50.777241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.777241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:50.777241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:50.777241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:50.777241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.777241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:50.777241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:50.777241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:50.777241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:50.778631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:50.778631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:50.778631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:50.778631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:50.778631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:50.778631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:50.778631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:50.778631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:50.778631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:50.778631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.778631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:50.778631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.779039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:50.779039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:50.779039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:50.779039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:50.779039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:50.779039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.779039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:50.779039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:50.779039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.779039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:50.779039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:50.779039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.779039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:50.779039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:50.779039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.779039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:50.779039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:50.779039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:50.779039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.779481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:50.780102 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:50.780102 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:50.780102 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:50.780102 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:50.780102 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:50.780102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.780668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:50.780668 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:50.780668 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:50.780668 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:50.780668 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:50.780668 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:50.780668 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.780668 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:50.780902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.780902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:50.780902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:50.780902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:50.780902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.780902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:50.780902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:50.780902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.780902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:50.780902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:50.780902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:50.780902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.781278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:50.781278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:50.781278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:50.781278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:50.781278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:50.781278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:50.781278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:50.781278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:50.781278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:50.781278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:50.781278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:50.781278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:50.781278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:50.782917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.782917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:50.782917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.782917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:50.782917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:50.782917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:50.782917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:50.782917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:50.782917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.782917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:50.782917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:50.782917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.782917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:50.782917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:50.782917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.783343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:50.783343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:50.783343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:50.783343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:50.783343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:50.783343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.783343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:50.783343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:50.783343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:50.783343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:50.783343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:50.783343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:50.783343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:50.783343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:50.783343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:50.783343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:50.783343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:50.783343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:50.783343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:50.783798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:50.783798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:50.783798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:50.783798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:50.783798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:50.783798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:53.669297 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:53.670037 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:53.670599 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:53.672097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:53.686580 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=235 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:29:53.723562 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:53.723720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:53.724165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:53.724165 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:53.724165 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:53.724165 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:53.724165 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:53.724165 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:53.724165 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:53.724165 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:53.725094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.053 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:53.725094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:53.725094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:53.725094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:53.725094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:53.725094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:53.725094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:53.725094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:53.725094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:53.725094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:53.725094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:53.725094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:53.725332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:53.725332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:53.725332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:53.725332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:53.725332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:53.725332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:53.725332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:53.725332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:53.725332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:53.725332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:53.725332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:53.725332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:53.725332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:53.725591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:53.725591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:53.725864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:53.725864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:53.725864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:53.725864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:53.725864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:53.725864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:53.725864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:53.725864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:53.725864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:53.725864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:53.725864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:53.725864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:53.725864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:53.725864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:53.725864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:53.725864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:53.726347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:53.726347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:53.726347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:53.726347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:53.726347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:53.726347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:53.726347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:53.726347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:53.726347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:53.726347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:53.726347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:53.726347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:53.726347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:53.726347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:53.726347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:53.726347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:53.726347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:53.726347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:53.726347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:53.726722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:53.726722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:53.726722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:55.739492 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:55.739850 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:55.739850 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:55.741834 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:55.741834 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:55.741834 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:55.741834 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:55.741834 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:55.742018 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:55.742018 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:55.742018 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:55.742018 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:55.742018 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:55.742018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.742163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:55.742163 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:55.742163 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:55.742163 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:55.742163 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:55.742163 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:55.742163 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.742163 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:55.742584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.742584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:55.742584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:55.742584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:55.742584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.742584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:55.742584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:55.742584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.742584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:55.742584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:55.742584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:55.742584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.742808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:55.742808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:55.742808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:55.742808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:55.742808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:55.742808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:55.742808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:55.742808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:55.742808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:55.742808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:55.742808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:55.742808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:55.742808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:55.743035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.743035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:55.743412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.743412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:55.743412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:55.743412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:55.743412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:55.743412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:55.743412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.743412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:55.743412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:55.743412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.743412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:55.743412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:55.743412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.743412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:55.743412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:55.743412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.743953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:29:55.743953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:55.743953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.743953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:55.743953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:55.743953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:55.743953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.743953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:55.743953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:55.743953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:55.743953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:55.743953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:55.743953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:55.743953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:55.743953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:55.743953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:55.743953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:55.743953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:55.743953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:55.744364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:55.744364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.744364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:55.750357 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:55.750843 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:55.750843 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:55.750843 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:29:55.751147 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:55.752511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.752969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:55.752969 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:55.752969 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:55.752969 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:55.752969 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:55.752969 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:55.752969 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.752969 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:55.753591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.753591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:55.753591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:55.753591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:55.753591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.753591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:55.753591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:55.753591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:55.753591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.753591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:55.753591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:55.753591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:55.753591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:55.753591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:55.753591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:55.753842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:55.753842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:55.753842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:55.753842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:55.753842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:55.753842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:55.753842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:55.753842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.753842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:55.754165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.754165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:55.754165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:55.754165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:55.754165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.754165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:55.754165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:55.754165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:29:55.754165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:55.754165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:55.754165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.754165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:55.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:55.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:55.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:55.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:55.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:55.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:55.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:55.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:55.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:55.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:55.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:55.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:55.754418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:55.754733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:55.754733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:55.754733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.754733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:55.754839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.754839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:55.754839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:55.754839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:55.754839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:55.754839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:55.754839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.754839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:55.754839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:55.754839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.754839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:55.754839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:55.754839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.754839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:55.754839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:55.754839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.755112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:29:55.755112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:29:55.755112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:29:55.755112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:55.755112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:55.755112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.755112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:55.755112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:55.755112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:55.755112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.755112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:55.755112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:55.755112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:55.755112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:55.755112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:55.755112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:55.755112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:55.755112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:55.755434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:55.755434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:55.755434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:55.755434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:55.755434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:55.755434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.755434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:55.765634 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:55.765969 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:55.766301 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:55.766491 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:55.767239 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:55.767239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.767554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:55.767554 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:55.767554 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:55.767554 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:55.767554 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:55.767554 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:55.767554 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.767554 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:55.768341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.768341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:29:55.768341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:55.768341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:55.768341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.768341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:55.768341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:55.768341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:29:55.768341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.768341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:29:55.768341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:55.768341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:29:55.768341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:55.768341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:29:55.768341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:29:55.768612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:55.768612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:29:55.768612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:55.768612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:29:55.768612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:29:55.768612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:55.768612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:29:55.768612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.768612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:29:55.769387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.769387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:55.769387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:55.769387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:55.769387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.769387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:55.769387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:55.769387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:55.769387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:55.769387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.769387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:55.769608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:55.769608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:55.769608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.769608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:55.769608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:55.769608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:55.769608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:55.769608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:55.769608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:55.769608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:55.769608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:55.769608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:55.769608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:55.769608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:55.769864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:55.769864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:55.769864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.769864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:55.770677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.770677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:55.770677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:55.770677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:55.770677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:55.770677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:55.770677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.770677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:55.770677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:55.770677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.770677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:55.770677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:55.770677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.770677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:55.770677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:55.770677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.770677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:29:55.771031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:29:55.771031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:29:55.771031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:29:55.771031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.771031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:55.771031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:55.771031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:55.771031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.771031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:55.771031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:55.771031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:55.771031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:55.771031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:55.771031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:55.771031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:55.771031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:55.771031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:55.771031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:55.771031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:55.771377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:55.771377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:55.771377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.771377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:55.777634 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:55.777861 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:55.777988 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:55.778172 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:55.778497 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:55.778732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:55.779051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.779546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:55.779546 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:55.779546 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:55.779546 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:55.779546 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:55.779546 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:55.779546 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.779546 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:55.780174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.780174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:55.780174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:55.780174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:55.780174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.780174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:55.780174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:55.780174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.780174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:55.780174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:55.780174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:55.780174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.780411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:55.780411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:55.780411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:55.780411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:55.780411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:55.780411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:55.780411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:55.780411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:55.780411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:55.780411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:55.780411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:55.780411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:55.780411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:55.780624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.780624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:55.780820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.780820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:55.780820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:55.780820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:55.780820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:55.780820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:55.780820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.780820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:55.780820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:55.780820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.780820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:55.780820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:55.780820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.780820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:55.780820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:55.780820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:55.781127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:29:55.781127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:55.781127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.781127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:55.781127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:55.781127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:55.781127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:55.781127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:55.781127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:55.781127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:55.781127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:55.781127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:55.781127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:55.781127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:55.781127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:55.781127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:55.781127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:55.781127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:55.781127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:55.781438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:55.781438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:55.781438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:29:58.670640 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:29:58.671095 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:29:58.671138 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:29:58.671734 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:29:58.718215 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:29:58.718380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:58.718884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:29:58.718884 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:58.718884 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:29:58.718884 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:29:58.718884 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:58.718884 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:29:58.718884 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:58.718884 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:29:58.720074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:58.720074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:29:58.720074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:58.720074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:29:58.720074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:58.720074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:58.720074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:58.720074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:58.720074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:58.720074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:58.720074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:29:58.720074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:58.720404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:29:58.720404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:58.720404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:29:58.720404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:58.720404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:29:58.720404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:29:58.720404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:58.720404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:29:58.720404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:58.720404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:29:58.720404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:29:58.720404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:58.720404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:29:58.720641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:58.720641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:29:58.721769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:58.721769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:29:58.721769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:29:58.721769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:29:58.721769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:29:58.721769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:29:58.721769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:58.721769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:29:58.721769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:29:58.721769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:58.721769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:29:58.721769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:29:58.721769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:29:58.721769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:29:58.721769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:29:58.721769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:29:58.722138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:29:58.722138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:29:58.722138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:58.722138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:58.722138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:29:58.722138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:29:58.722138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:29:58.722138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:29:58.722138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:29:58.722138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:29:58.722138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:29:58.722138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:29:58.722138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:29:58.722138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:29:58.722138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:29:58.722138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:29:58.722138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:29:58.722138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:29:58.722138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:29:58.722538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:29:58.722538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:29:58.722538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:00.740320 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:00.740320 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:00.740709 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:00.741402 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:00.741580 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:00.741697 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:00.741855 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:00.741926 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:00.742023 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:00.742113 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:00.742241 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:00.742463 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:00.742463 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:00.743024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.743272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:00.743272 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:00.743272 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:00.743272 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:00.743272 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:00.743272 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:00.743272 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.743272 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:00.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:00.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:00.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:00.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:00.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:00.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:00.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:00.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:00.746924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.747220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:00.747220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:00.747220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:00.747220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:00.747220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:00.747220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:00.747220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:00.747220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:00.747220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:00.747220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:00.747220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:00.747220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:00.747220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:00.747458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.747458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:00.747458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.747458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:00.747458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:00.747458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:00.747458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:00.747458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:00.747458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.747458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:00.747458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:00.747458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.747458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:00.747458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:00.747458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:00.747715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:00.748045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:00.748045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:00.748045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:00.748045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:00.748045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.748045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:00.751360 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:00.751360 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:00.751360 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:00.751360 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:30:00.752268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:00.752268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.752581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:00.752581 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:00.752581 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:00.752581 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:00.752581 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:00.752581 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:00.752581 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.752581 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:00.753215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.753215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:00.753215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:00.753215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:00.753215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.753215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:00.753215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:00.753215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:00.753215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.753215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:00.753215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:00.753215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:00.753215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:00.753215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:00.753215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:00.754196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:00.754196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:00.754196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:00.754196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:00.754196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:00.754196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:00.754196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:00.754196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.754196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:00.754196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.754196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:00.754196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:00.754196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:00.754196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.754466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:00.754466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:00.754466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:30:00.754466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:00.754466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:00.754466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.754466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:00.754466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:00.754466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:00.754466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.754466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:00.754466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:00.754466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:00.754696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:00.754696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:00.754696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:00.754696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:00.754696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:00.754696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:00.754696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:00.754696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:00.754696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:00.754696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:00.754696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.754696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:00.754914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.754914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:00.754914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:00.754914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:00.754914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:00.754914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:00.754914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.754914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:00.754914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:00.754914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.754914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:00.754914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:00.754914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.754914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:00.754914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:00.754914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.755206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:00.755206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:00.755206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:30:00.755206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:00.755206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:00.755206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.755206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:00.755206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:00.755206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:00.755206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.755206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:00.755206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:00.755206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:00.755206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:00.755206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:00.755206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:00.755206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:00.755206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:00.755520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:00.755520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:00.755520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:00.755520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:00.755520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:00.755520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.755520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:00.766985 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:00.766985 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:00.766985 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:00.766985 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:00.767467 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:00.767671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.768189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:00.768189 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:00.768189 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:00.768189 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:00.768189 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:00.768189 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:00.768189 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.768189 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:00.769176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.769176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:00.769176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:00.769176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:00.769176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.769176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:00.769176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:00.769176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:00.769176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.769176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:00.769176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:00.769176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:00.769176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:00.769176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:00.769176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:00.769821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:00.769821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:00.769821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:00.769821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:00.769821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:00.769821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:00.769821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:00.769821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.769821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:00.770347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.770347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:00.770347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:00.770347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:00.770347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.770347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:00.770347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:00.770347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:00.770347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:00.770347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.770347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:00.771403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:00.771403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:00.771403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.771403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:00.771403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:00.771403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:00.771403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:00.771403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:00.771403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:00.771403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:00.771403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:00.771403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:00.771403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:00.771403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:00.771917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:00.771917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:00.771917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.771917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:00.772329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.772329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:00.772329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:00.772329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:00.772329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:00.772329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:00.772329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.772329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:00.772329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:00.772329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.772329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:00.772329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:00.772329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.772329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:00.772329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:00.772329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.772329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:00.773020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:00.773020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:00.773020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:00.773020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.773020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:00.773020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:00.773020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:00.773020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.773020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:00.773020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:00.773020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:00.773020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:00.773020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:00.773020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:00.773020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:00.773020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:00.773020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:00.773020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:00.773020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:00.773717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:00.773717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:00.773717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.773717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:00.780663 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:00.781095 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:00.781303 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:00.781516 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:00.781899 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:00.782110 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:00.782380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.782852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:00.782852 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:00.782852 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:00.782852 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:00.782852 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:00.782852 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:00.782852 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.782852 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:00.784168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.784168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:00.784168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:00.784168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:00.784168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.784168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:00.784168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:00.784168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.784168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:00.784168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:00.784168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:00.784168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:00.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:00.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:00.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:00.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:00.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:00.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:00.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:00.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:00.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:00.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:00.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:00.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:00.784691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.784691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:00.787117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.787117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:00.787117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:00.787117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:00.787117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:00.787117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:00.787117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.787117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:00.787117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:00.787117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.787117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:00.787117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:00.787117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.787117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:00.787117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:00.787117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:00.787449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:00.787449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:00.787449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.787449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:00.787449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:00.787449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:00.787449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:00.787449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:00.787449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:00.787449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:00.787449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:00.787449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:00.787449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:00.787449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:00.787449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:00.787449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:00.787449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:00.787449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:00.787449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:00.787755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:00.787755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:00.787755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:03.671109 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:03.671496 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:03.671496 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:03.671963 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:03.713997 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:03.714247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:03.714779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:03.714779 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:03.714779 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:03.714779 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:03.714779 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:03.714779 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:03.714779 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:03.714779 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:03.715516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:03.715516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:03.715516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:03.715516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:03.715516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:03.715516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:03.715516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:03.715516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:03.715516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:03.715516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:03.715516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:03.715516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:03.715808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:03.715808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:03.715808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:03.715808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:03.715808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:03.715808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:03.715808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:03.715808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:03.715808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:03.715808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:03.715808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:03.715808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:03.715808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:03.716102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:03.716102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:03.716235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:03.716235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:03.716235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:03.716235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:03.716235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:03.716235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:03.716235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:03.716235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:03.716235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:03.716235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:03.716235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:03.716235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:03.716235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:03.716235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:03.716235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:03.716235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:03.716573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:03.716573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:03.716573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:03.716573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:03.716573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:03.716573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:03.716573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:03.716573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:03.716573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:03.716573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:03.716573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:03.716573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:03.716573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:03.716573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:03.716573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:03.716573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:03.716573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:03.716573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:03.716573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:03.716924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:03.716924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:03.716924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:05.740358 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:05.740776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:05.740776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:05.742144 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:05.742344 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:05.742462 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:05.742571 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:05.742671 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:05.742768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:05.742865 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:05.742960 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:05.743065 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:05.743194 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:05.743313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.743786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:05.743786 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:05.743786 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:05.743786 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:05.743786 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:05.743786 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:05.743786 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.743786 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:05.744880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.744880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:05.744880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:05.744880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:05.744880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.744880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:05.744880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:05.744880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.744880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:05.744880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:05.744880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:05.744880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.745124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:05.745124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:05.745124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:05.745124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:05.745124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:05.745124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:05.745124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:05.745124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:05.745124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:05.745124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:05.745124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:05.745124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:05.745124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:05.745354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.745354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:05.746046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.746046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:05.746046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:05.746046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:05.746046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:05.746046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:05.746046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.746046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:05.746046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:05.746046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.746046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:05.746046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:05.746046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.746046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:05.746046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:05.746046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.746357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:05.746357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:05.746357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.746357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:05.746357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:05.746357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:05.746357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.746357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:05.746357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:05.746357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:05.746357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:05.746357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:05.746357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:05.746357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:05.746357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:05.746357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:05.746357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:05.746357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:05.746357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:05.746711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:05.746711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.746711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:05.751026 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:05.751295 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:05.752449 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:05.752449 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:30:05.752449 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:05.752449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.753106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:05.753106 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:05.753106 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:05.753106 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:05.753106 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:05.753106 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:05.753106 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.753106 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:05.753620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.753620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:05.753620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:05.753620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:05.753620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.753620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:05.753620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:05.753620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:05.753620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.753620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:05.753620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:05.753620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:05.753620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:05.753620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:05.753620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:05.753881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:05.753881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:05.753881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:05.753881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:05.753881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:05.753881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:05.753881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:05.753881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.753881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:05.754526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.754526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:05.754526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:05.754526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:05.754526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.754526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:05.754526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:05.754526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:30:05.754526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:05.754526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:05.754526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.754526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:05.754738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:05.754738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:05.754738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.754738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:05.754738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:05.754738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:05.754738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:05.754738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:05.754738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:05.754738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:05.754738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:05.754738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:05.754738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:05.754738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:05.754960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:05.754960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:05.754960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.754960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:05.755657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.755657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:05.755657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:05.755657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:05.755657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:05.755657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:05.755657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.755657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:05.755657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:05.755657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.755657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:05.755657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:05.755657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.755657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:05.755657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:05.755657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:05.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:05.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:30:05.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:05.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:05.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:05.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:05.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:05.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:05.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:05.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:05.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:05.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:05.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:05.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:05.755979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:05.756309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:05.756309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:05.756309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:05.756309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:05.756309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:05.756309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.756309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:05.766566 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:05.766835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:05.767027 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:05.767371 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:05.767899 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:05.768134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.768647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:05.768647 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:05.768647 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:05.768647 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:05.768647 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:05.768647 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:05.768647 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.768647 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:05.769555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.769555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:05.769555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:05.769555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:05.769555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.769555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:05.769555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:05.769555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:05.769555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.769555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:05.769555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:05.769555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:05.769555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:05.769555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:05.769555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:05.769837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:05.769837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:05.769837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:05.769837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:05.769837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:05.769837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:05.769837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:05.769837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.769837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:05.770508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.770508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:05.770508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:05.770508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:05.770508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.770508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:05.770508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:05.770508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:05.770508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:05.770508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.770508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:05.770732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:05.770732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:05.770732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.770732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:05.770732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:05.770732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:05.770732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:05.770732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:05.770732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:05.770732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:05.770732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:05.770732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:05.770732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:05.770732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:05.770956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:05.770956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:05.770956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.770956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:05.771669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.771669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:05.771669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:05.771669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:05.771669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:05.771669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:05.771669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.771669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:05.771669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:05.771669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.771669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:05.771669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:05.771669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.771669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:05.771669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:05.771669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.771669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:05.772079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:05.772079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:05.772079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:05.772079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.772079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:05.772079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:05.772079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:05.772079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.772079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:05.772079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:05.772079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:05.772079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:05.772079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:05.772079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:05.772079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:05.772079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:05.772079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:05.772079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:05.772079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:05.772418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:05.772418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:05.772418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.772418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:05.781194 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:05.781489 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:05.781627 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:05.781853 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:05.782354 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:05.782637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:05.782922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.783464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:05.783464 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:05.783464 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:05.783464 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:05.783464 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:05.783464 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:05.783464 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.783464 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:05.784174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.784174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:05.784174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:05.784174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:05.784174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.784174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:05.784174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:05.784174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.784174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:05.784174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:05.784174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:05.784174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.784435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:05.784435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:05.784435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:05.784435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:05.784435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:05.784435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:05.784435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:05.784435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:05.784435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:05.784435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:05.784435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:05.784435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:05.784435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:05.784641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.784641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:05.784806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.784806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:05.784806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:05.784806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:05.784806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:05.784806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:05.784806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.784806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:05.784806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:05.784806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.784806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:05.784806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:05.784806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.784806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:05.784806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:05.784806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:05.785101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:05.785101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:05.785101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.785101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:05.785101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:05.785101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:05.785101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:05.785101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:05.785101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:05.785101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:05.785101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:05.785101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:05.785101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:05.785101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:05.785101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:05.785101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:05.785101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:05.785101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:05.785101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:05.785479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:05.785479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:05.785479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:08.671549 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:08.672223 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:08.672223 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:08.672663 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:08.701215 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:08.701403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:08.701855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:08.701855 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:08.701855 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:08.701855 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:08.701855 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:08.701855 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:08.701855 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:08.701855 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:08.702712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:08.702712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:08.702712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:08.702712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:08.702712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:08.702712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:08.702712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:08.702712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:08.702712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:08.702712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:08.702712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:08.702712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:08.703148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:08.703148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:08.703148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:08.703148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:08.703148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:08.703148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:08.703148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:08.703148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:08.703148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:08.703148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:08.703148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:08.703148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:08.703148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:08.703616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:08.703616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:08.703821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:08.703821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:08.703821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:08.703821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:08.703821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:08.703821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:08.703821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:08.703821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:08.703821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:08.703821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:08.703821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:08.703821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:08.703821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:08.703821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:08.703821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:08.703821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:08.704379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:08.704379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:08.704379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:08.704379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:08.704379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:08.704379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:08.704379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:08.704379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:08.704379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:08.704379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:08.704379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:08.704379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:08.704379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:08.704379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:08.704379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:08.704379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:08.704379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:08.704379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:08.704379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:08.704979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:08.704979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:08.704979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:10.740809 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:10.741329 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:10.741329 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:10.742393 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:10.742610 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:10.742776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:10.742925 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:10.743058 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:10.743165 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:10.743294 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:10.743387 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:10.743499 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:10.743610 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:10.743724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.744190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:10.744190 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:10.744190 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:10.744190 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:10.744190 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:10.744190 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:10.744190 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.744190 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:10.744894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.744894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:10.744894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:10.744894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:10.744894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.744894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:10.744894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:10.744894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.744894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:10.744894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:10.744894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:10.744894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.745293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:10.745293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:10.745293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:10.745293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:10.745293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:10.745293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:10.745293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:10.745293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:10.745293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:10.745293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:10.745293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:10.745293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:10.745293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:10.745586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.745586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:10.745754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.745754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:10.745754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:10.745754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:10.745754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:10.745754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:10.745754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.745754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:10.745754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:10.745754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.745754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:10.745754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:10.745754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.745754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:10.745754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:10.745754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.746202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:10.746202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:10.746202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.746202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:10.746202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:10.746202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:10.746202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.746202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:10.746202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:10.746202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:10.746202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:10.746202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:10.746202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:10.746202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:10.746202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:10.746202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:10.746202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:10.746202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:10.746202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:10.746618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:10.746618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.746618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:10.751353 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:10.751604 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:10.751698 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:10.751788 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:30:10.752133 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:10.752339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.752702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:10.752702 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:10.752702 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:10.752702 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:10.752702 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:10.752702 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:10.752702 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.752702 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:10.753448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.753448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:10.753448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:10.753448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:10.753448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.753448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:10.753448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:10.753448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:10.753448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.753448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:10.753448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:10.753448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:10.753448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:10.753448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:10.753448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:10.753782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:10.753782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:10.753782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:10.753782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:10.753782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:10.753782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:10.753782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:10.753782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.753782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:10.754081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.754081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:10.754081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:10.754081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:10.754081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.754081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:10.754081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:10.754081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:30:10.754081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:10.754081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:10.754081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.754081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:10.754445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:10.754445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:10.754445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.754445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:10.754445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:10.754445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:10.754445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:10.754445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:10.754445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:10.754445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:10.754445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:10.754445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:10.754445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:10.754445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:10.754807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:10.754807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:10.754807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.754807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:10.754916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.754916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:10.754916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:10.754916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:10.754916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:10.754916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:10.754916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.754916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:10.754916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:10.754916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.754916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:10.754916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:10.754916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.754916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:10.754916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:10.754916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.755327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:10.755327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:10.755327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:30:10.755327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:10.755327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:10.755327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.755327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:10.755327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:10.755327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:10.755327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.755327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:10.755327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:10.755327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:10.755327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:10.755327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:10.755327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:10.755327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:10.755327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:10.755747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:10.755747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:10.755747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:10.755747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:10.755747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:10.755747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.755747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:10.766873 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:10.767071 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:10.767170 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:10.767339 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:10.767604 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:10.767802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.768223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:10.768223 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:10.768223 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:10.768223 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:10.768223 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:10.768223 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:10.768223 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.768223 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:10.768887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.768887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:10.768887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:10.768887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:10.768887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.768887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:10.768887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:10.768887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:10.768887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.768887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:10.768887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:10.768887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:10.768887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:10.768887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:10.768887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:10.769332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:10.769332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:10.769332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:10.769332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:10.769332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:10.769332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:10.769332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:10.769332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.769332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:10.769613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.769613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:10.769613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:10.769613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:10.769613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.769613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:10.769613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:10.769613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:10.769613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:10.769613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.769613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:10.770030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:10.770030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:10.770030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.770030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:10.770030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:10.770030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:10.770030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:10.770030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:10.770030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:10.770030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:10.770030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:10.770030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:10.770030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:10.770030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:10.770453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:10.770453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:10.770453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.770453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:10.770453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.770453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:10.770453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:10.770453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:10.770453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:10.770453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:10.770453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.770453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:10.770453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:10.770453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.770891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:10.770891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:10.770891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.770891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:10.770891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:10.770891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.770891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:10.770891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:10.770891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:10.770891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:10.770891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.770891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:10.770891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:10.770891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:10.770891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.770891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:10.770891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:10.770891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:10.770891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:10.771562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:10.771562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:10.771562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:10.771562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:10.771562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:10.771562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:10.771562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:10.771562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:10.771562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:10.771562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.771562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:10.781578 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:10.781770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:10.781848 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:10.781984 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:10.782278 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:10.782505 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:10.782779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.783123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:10.783123 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:10.783123 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:10.783123 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:10.783123 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:10.783123 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:10.783123 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.783123 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:10.783734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.783734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:10.783734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:10.783734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:10.783734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.783734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:10.783734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:10.783734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.783734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:10.783734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:10.783734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:10.783734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:10.784574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.784574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:10.784574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.784574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:10.784574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:10.784574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:10.784574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:10.784574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:10.784574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.784574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:10.784574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:10.784574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.784574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:10.784574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:10.784574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.785078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:10.785078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:10.785078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:10.785078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:10.785078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:10.785078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.785078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:10.785078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:10.785078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:10.785078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:10.785078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:10.785078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:10.785078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:10.785078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:10.785078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:10.785078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:10.785078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:10.785078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:10.785078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:10.785814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:10.785814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:10.785814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:10.785814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:10.785814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:10.785814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:13.672024 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:13.672477 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:13.672477 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:13.672839 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:13.717691 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:13.717895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:13.719947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:13.719947 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:13.719947 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:13.719947 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:13.719947 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:13.719947 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:13.719947 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:13.719947 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:13.719947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:13.719947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:13.719947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:13.719947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:13.719947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:13.719947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:13.719947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:13.720343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:13.720343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:13.720343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:13.720343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:13.720343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:13.720343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:13.720343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:13.720343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:13.720343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:13.720343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:13.720343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:13.720343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:13.720343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:13.720343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:13.720590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:13.720590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:13.720590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:13.720590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:13.720590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:13.720590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:13.720590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:13.720590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:13.720590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:13.720590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:13.720590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:13.720590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:13.720590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:13.720590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:13.720855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:13.721274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:13.721274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:13.721274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:13.721274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:13.721274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:13.721274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:13.721274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:13.721274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:13.721274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:15.741267 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:15.742768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:15.742909 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:15.744343 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:15.744551 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:15.744825 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:15.744825 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:15.744948 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:15.745072 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:15.745211 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:15.745342 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:15.745447 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:15.745547 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:15.745650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.747465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:15.747465 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:15.747465 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:15.747465 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:15.747465 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:15.747465 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:15.747465 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.747465 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:15.748508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.748508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:15.748508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:15.748508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:15.748508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.748508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:15.748508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:15.748508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.748508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:15.748508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:15.748508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:15.748508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.748944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:15.748944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:15.748944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:15.748944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:15.748944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:15.748944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:15.748944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:15.748944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:15.748944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:15.748944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:15.748944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:15.748944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:15.748944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:15.749624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.749624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:15.750933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.750933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:15.750933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:15.750933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:15.750933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:15.750933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:15.750933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.750933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:15.750933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:15.750933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.750933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:15.750933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:15.750933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.750933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:15.750933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:15.750933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.755150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:15.755150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:15.755150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.755150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:15.755150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:15.755150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:15.755150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.755150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:15.755150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:15.755150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:15.755150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:15.755150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:15.755150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:15.755150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:15.755150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:15.755150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:15.755150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:15.755150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:15.755150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:15.755896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:15.755896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.755896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:15.755896 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:15.755896 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:15.755896 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:15.755896 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:30:15.755896 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:15.756241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.756241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:15.756241 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:15.756241 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:15.756241 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:15.756241 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:15.756241 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:15.756241 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.756241 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:15.756241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.756241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:15.756241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:15.756241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:15.756241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.756241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:15.756241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:15.756241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:15.756878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.756878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:15.756878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:15.756878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:15.756878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:15.756878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:15.756878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:15.756878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:15.756878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:15.756878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:15.756878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:15.756878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:15.756878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:15.756878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:15.756878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.756878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:15.757497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.757497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:15.757497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:15.757497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:15.757497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.757497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:15.757497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:15.757497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:30:15.757497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:15.757497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:15.757497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.757497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:15.758067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:15.758067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:15.758067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.758067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:15.758067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:15.758067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:15.758067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:15.758067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:15.758067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:15.758067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:15.758067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:15.758067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:15.758067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:15.758067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:15.758464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:15.758464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:15.758464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.758464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:15.758464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.758464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:15.758464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:15.758464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:15.758464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:15.758464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:15.758464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.758464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:15.758464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:15.758464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.758702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:15.758702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:15.758702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.758702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:15.758702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:15.758702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.758702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:15.758702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:15.758702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:30:15.758702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:15.758702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:15.758702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.758702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:15.758702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:15.758702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:15.758702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.758702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:15.758702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:15.758702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:15.759605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:15.759605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:15.759605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:15.759605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:15.759605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:15.759605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:15.759605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:15.759605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:15.759605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:15.759605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:15.759605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.759605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:15.767283 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:15.770812 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:15.771081 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:15.771081 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:15.771482 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:15.771799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.772388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:15.772388 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:15.772388 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:15.772388 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:15.772388 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:15.772388 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:15.772388 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.772388 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:15.774191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.774191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:15.774191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:15.774191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:15.774191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.774191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:15.774191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:15.774191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:15.774191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.774191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:15.774191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:15.774191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:15.774191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:15.774191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:15.774191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:15.774493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:15.774493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:15.774493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:15.774493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:15.774493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:15.774493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:15.774493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:15.774493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.774493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:15.774921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.774921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:15.774921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:15.774921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:15.774921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.774921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:15.774921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:15.774921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:15.774921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:15.774921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.774921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:15.775392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:15.775392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:15.775392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.775392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:15.775392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:15.775392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:15.775392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:15.775392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:15.775392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:15.775392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:15.775392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:15.775392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:15.775392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:15.775392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:15.775822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:15.775822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:15.775822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.775822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:15.775822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.775822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:15.775822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:15.775822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:15.775822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:15.775822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:15.775822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.775822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:15.775822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:15.775822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.776093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:15.776093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:15.776093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.776093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:15.776093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:15.776093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.776093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:15.776093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:15.776093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:15.776093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:15.776093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.776093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:15.776093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:15.776093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:15.776093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.776093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:15.776093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:15.776093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:15.776093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:15.776426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:15.776426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:15.776426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:15.776426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:15.776426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:15.776426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:15.776426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:15.776426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:15.776426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:15.776426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.776426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:15.781956 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:15.782830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:15.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:15.783094 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:15.783620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:15.783916 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:15.784299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.784874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:15.784874 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:15.784874 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:15.784874 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:15.784874 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:15.784874 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:15.784874 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.784874 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:15.785591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.785591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:15.785591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:15.785591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:15.785591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.785591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:15.785591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:15.785591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.785591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:15.785591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:15.785591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:15.785591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.785825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:15.785825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:15.785825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:15.785825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:15.785825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:15.785825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:15.785825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:15.785825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:15.785825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:15.785825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:15.785825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:15.785825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:15.785825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:15.786056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.786056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:15.786265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.786265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:15.786265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:15.786265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:15.786265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:15.786265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:15.786265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.786265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:15.786265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:15.786265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.786265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:15.786265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:15.786265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.786265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:15.786265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:15.786265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:15.786651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:15.786651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:15.786651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.786651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:15.786651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:15.786651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:15.786651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:15.786651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:15.786651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:15.786651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:15.786651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:15.786651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:15.786651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:15.786651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:15.786651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:15.786651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:15.786651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:15.786651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:15.786651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:15.787145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:15.787145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:15.787145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:18.672347 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:18.673371 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:18.673482 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:18.674204 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:18.728100 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:18.728286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:18.728801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:18.728801 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:18.728801 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:18.728801 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:18.728801 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:18.728801 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:18.728801 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:18.728801 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:18.729985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:18.729985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:18.729985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:18.729985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:18.729985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:18.729985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:18.729985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:18.729985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:18.729985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:18.729985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:18.729985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:18.729985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:18.730634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:18.730634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:18.730634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:18.730634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:18.730634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:18.730634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:18.730634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:18.730634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:18.730634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:18.730634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:18.730634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:18.730634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:18.730634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:18.731032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:18.731032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:18.731032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:18.731032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:18.731032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:18.731032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:18.731032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:18.731032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:18.731032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:18.731032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:18.731032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:18.731032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:18.731032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:18.731032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:18.731032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:18.731513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:18.731513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:18.731513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:18.731513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:18.731513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:18.731513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:18.731513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:18.731513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:18.731513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:18.731513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:18.731513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:18.731513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:18.731513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:18.731513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:18.731513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:18.731513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:18.731513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:18.731513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:18.731513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:18.732150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:18.732150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:18.732150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:18.732150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:18.732150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:18.732150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:20.742814 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:20.743336 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:20.743336 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:20.744476 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:20.744617 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:20.744704 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:20.744842 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:20.744954 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:20.745089 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:20.745227 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:20.745365 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:20.745468 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:20.745561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:20.745660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.746183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:20.746183 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:20.746183 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:20.746183 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:20.746183 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:20.746183 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:20.746183 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.746183 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:20.750274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.750274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:20.750274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:20.750274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:20.750274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.750274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:20.750274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:20.750274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.750274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:20.750274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:20.750274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:20.750274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.750652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:20.750652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:20.750652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:20.750652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:20.750652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:20.750652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:20.750652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:20.750652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:20.750652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:20.750652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:20.750652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:20.750652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:20.750652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:20.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:20.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:20.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:20.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:20.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:20.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:20.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:20.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:20.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:20.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:20.750900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.751279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:20.751279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:20.751279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.751279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:20.751279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:20.751279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.751279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:20.751279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:20.751279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:20.751279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.751279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:20.751279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:20.751279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:20.751279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:20.751279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:20.751279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:20.751279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:20.751279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:20.751279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:20.751673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:20.751673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:20.751673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:20.751673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:20.751673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.751673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:20.753583 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:20.753923 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:20.754027 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:20.754125 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:30:20.754558 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:20.754744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.755204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:20.755204 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:20.755204 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:20.755204 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:20.755204 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:20.755204 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:20.755204 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.755204 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:20.755826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.755826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:20.755826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:20.755826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:20.755826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.755826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:20.755826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:20.755826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:20.755826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.755826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:20.755826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:20.755826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:20.755826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:20.755826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:20.755826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:20.756177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:20.756177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:20.756177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:20.756177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:20.756177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:20.756177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:20.756177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:20.756177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.756177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:20.756446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.756446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:20.756446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:20.756446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:20.756446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.756446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:20.756446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:20.756446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:30:20.756446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:20.756446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:20.756446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.756446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:20.756741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:20.756741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:20.756741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.756741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:20.756741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:20.756741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:20.756741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:20.756741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:20.756741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:20.756741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:20.756741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:20.756741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:20.756741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:20.756741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:20.757023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:20.757023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:20.757023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.757023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:20.757219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.757219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:20.757219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:20.757219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:20.757219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:20.757219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:20.757219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.757219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:20.757219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:20.757219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.757219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:20.757219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:20.757219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.757219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:20.757219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:20.757219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.757619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:20.757619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:20.757619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:30:20.757619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:20.757619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:20.757619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.757619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:20.757619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:20.757619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:20.757619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.757619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:20.757619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:20.757619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:20.757619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:20.757619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:20.757619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:20.757619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:20.757619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:20.757963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:20.757963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:20.757963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:20.757963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:20.757963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:20.757963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.757963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:20.770819 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:20.771180 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:20.771346 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:20.771474 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:20.771929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:20.772187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.772721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:20.772721 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:20.772721 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:20.772721 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:20.772721 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:20.772721 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:20.772721 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.772721 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:20.773387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.773387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:20.773387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:20.773387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:20.773387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.773387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:20.773387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:20.773387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:20.773387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.773387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:20.773387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:20.773387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:20.773387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:20.773387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:20.773387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:20.773748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:20.773748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:20.773748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:20.773748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:20.773748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:20.773748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:20.773748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:20.773748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.773748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:20.773985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.773985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:20.773985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:20.773985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:20.773985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.773985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:20.773985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:20.773985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:20.773985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:20.773985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.773985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:20.774273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:20.774273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:20.774273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.774273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:20.774273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:20.774273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:20.774273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:20.774273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:20.774273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:20.774273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:20.774273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:20.774273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:20.774273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:20.774273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:20.774555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:20.774555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:20.774555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.774555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:20.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:20.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:20.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:20.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:20.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:20.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:20.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:20.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:20.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:20.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:20.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:20.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:20.775185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:20.775185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:20.775185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:20.775185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.775185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:20.775185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:20.775185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:20.775185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.775185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:20.775185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:20.775185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:20.775185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:20.775185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:20.775185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:20.775185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:20.775185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:20.775185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:20.775185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:20.775185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:20.775554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:20.775554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:20.775554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.775554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:20.782836 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:20.783169 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:20.783312 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:20.783471 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:20.783885 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:20.784122 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:20.784403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.784878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:20.784878 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:20.784878 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:20.784878 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:20.784878 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:20.784878 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:20.784878 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.784878 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:20.785607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.785607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:20.785607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:20.785607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:20.785607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.785607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:20.785607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:20.785607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.785607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:20.785607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:20.785607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:20.785607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.785945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:20.785945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:20.785945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:20.785945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:20.785945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:20.785945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:20.785945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:20.785945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:20.785945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:20.785945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:20.785945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:20.785945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:20.785945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:20.786197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.786197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:20.787317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.787317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:20.787317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:20.787317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:20.787317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:20.787317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:20.787317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.787317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:20.787317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:20.787317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.787317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:20.787317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:20.787317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.787317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:20.787317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:20.787317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:20.787776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:20.787776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:20.787776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.787776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:20.787776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:20.787776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:20.787776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:20.787776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:20.787776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:20.787776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:20.787776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:20.787776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:20.787776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:20.787776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:20.787776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:20.787776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:20.787776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:20.787776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:20.787776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:20.788156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:20.788156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:20.788156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:24.191426 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:24.191426 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:24.191426 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:24.191426 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:24.191426 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:24.191426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:24.195962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:24.195962 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:24.195962 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:24.195962 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:24.195962 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:24.195962 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:24.195962 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:24.195962 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:24.195962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:24.195962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:24.195962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:24.195962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:24.195962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:24.195962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:24.195962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:24.197449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:24.197449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:24.197449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:24.197449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:24.197449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:24.197449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:24.197449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:24.197449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:24.197449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:24.197449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:24.197449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:24.197449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:24.197449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:24.197449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:24.198278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:24.198278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:24.198278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:24.198278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:24.198278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:24.198278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:24.198278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:24.198278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:24.198278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:24.198278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:24.198278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:24.198278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:24.198278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:24.198278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:24.198951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:24.199711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:24.199711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:24.199711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:24.199711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:24.199711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:24.199711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:24.199711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:24.199711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:24.199711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:25.743303 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:25.743833 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:25.743879 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:25.744867 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:25.745051 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:25.745191 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:25.745343 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:25.745477 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:25.745602 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:25.745726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:25.745852 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:25.745972 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:25.746119 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:25.746242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.746765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:25.746765 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:25.746765 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:25.746765 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:25.746765 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:25.746765 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:25.746765 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.746765 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:25.747540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.747540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:25.747540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:25.747540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:25.747540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.747540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:25.747540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:25.747540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.747540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:25.747540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:25.747540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:25.747540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.747859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:25.747859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:25.747859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:25.747859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:25.747859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:25.747859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:25.747859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:25.747859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:25.747859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:25.747859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:25.747859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:25.747859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:25.747859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:25.748187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.748187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:25.748540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.748540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:25.748540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:25.748540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:25.748540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:25.748540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:25.748540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.748540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:25.748540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:25.748540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.748540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:25.748540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:25.748540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.748540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:25.748540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:25.748540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.748933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:25.748933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:25.748933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.748933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:25.748933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:25.748933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:25.748933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.748933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:25.748933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:25.748933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:25.748933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:25.748933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:25.748933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:25.748933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:25.748933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:25.748933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:25.748933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:25.748933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:25.748933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:25.749410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:25.749410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.749410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:25.753891 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:25.754298 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:25.754425 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:25.754579 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:30:25.754950 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:25.755119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.755609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:25.755609 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:25.755609 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:25.755609 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:25.755609 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:25.755609 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:25.755609 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.755609 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:25.756292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.756292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:25.756292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:25.756292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:25.756292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.756292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:25.756292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:25.756292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:25.756292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.756292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:25.756292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:25.756292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:25.756292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:25.756292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:25.756292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:25.756668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:25.756668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:25.756668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:25.756668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:25.756668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:25.756668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:25.756668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:25.756668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.756668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:25.757114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.757114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:25.757114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:25.757114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:25.757114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.757114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:25.757114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:25.757114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:30:25.757114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:25.757114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:25.757114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.757114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:25.757451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:25.757451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:25.757451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.757451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:25.757451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:25.757451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:25.757451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:25.757451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:25.757451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:25.757451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:25.757451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:25.757451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:25.757451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:25.757451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:25.757786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:25.757786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:25.757786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.757786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:25.758144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.758144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:25.758144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:25.758144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:25.758144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:25.758144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:25.758144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.758144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:25.758144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:25.758144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.758144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:25.758144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:25.758144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.758144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:25.758144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:25.758144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.758557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:25.758557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:25.758557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:30:25.758557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:25.758557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:25.758557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.758557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:25.758557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:25.758557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:25.758557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.758557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:25.758557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:25.758557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:25.758557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:25.758557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:25.758557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:25.758557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:25.758557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:25.758962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:25.758962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:25.758962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:25.758962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:25.758962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:25.758962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.758962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:25.771200 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:25.771571 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:25.771742 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:25.771924 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:25.772317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:25.772542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.773514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:25.773514 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:25.773514 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:25.773514 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:25.773514 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:25.773514 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:25.773514 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.773514 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:25.773779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.773779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:25.773779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:25.773779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:25.773779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.773779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:25.773779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:25.773779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:25.773779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.773779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:25.773779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:25.773779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:25.773779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:25.773779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:25.773779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:25.774160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:25.774160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:25.774160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:25.774160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:25.774160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:25.774160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:25.774160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:25.774160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.774160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:25.774929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.774929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:25.774929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:25.774929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:25.774929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.774929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:25.774929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:25.774929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:25.774929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:25.774929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.774929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:25.775232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:25.775232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:25.775232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.775232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:25.775232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:25.775232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:25.775232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:25.775232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:25.775232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:25.775232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:25.775232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:25.775232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:25.775232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:25.775232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:25.775602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:25.775602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:25.775602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.775602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:25.776167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.776167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:25.776167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:25.776167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:25.776167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:25.776167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:25.776167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.776167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:25.776167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:25.776167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.776167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:25.776167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:25.776167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.776167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:25.776167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:25.776167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.776167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:25.776631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:25.776631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:25.776631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:25.776631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.776631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:25.776631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:25.776631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:25.776631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.776631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:25.776631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:25.776631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:25.776631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:25.776631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:25.776631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:25.776631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:25.776631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:25.776631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:25.776631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:25.776631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:25.777056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:25.777056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:25.777056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.777056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:25.783154 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:25.783411 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:25.783543 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:25.783707 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:25.784010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:25.784261 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:25.784554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.784999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:25.784999 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:25.784999 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:25.784999 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:25.784999 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:25.784999 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:25.784999 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.784999 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:25.785660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.785660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:25.785660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:25.785660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:25.785660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.785660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:25.785660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:25.785660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.785660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:25.785660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:25.785660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:25.785660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.785976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:25.785976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:25.785976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:25.785976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:25.785976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:25.785976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:25.785976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:25.785976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:25.785976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:25.785976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:25.785976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:25.785976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:25.785976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:25.786321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.786321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:25.786635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.786635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:25.786635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:25.786635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:25.786635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:25.786635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:25.786635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.786635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:25.786635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:25.786635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.786635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:25.786635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:25.786635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.786635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:25.786635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:25.786635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:25.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:25.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:25.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:25.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:25.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:25.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:25.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:25.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:25.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:25.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:25.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:25.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:25.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:25.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:25.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:25.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:25.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:25.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:25.787487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:25.787487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:25.787487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:28.673945 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:28.674403 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:28.674403 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:28.674896 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:28.706416 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:28.706594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:28.707113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:28.707113 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:28.707113 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:28.707113 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:28.707113 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:28.707113 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:28.707113 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:28.707113 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:28.707838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:28.707838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:28.707838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:28.707838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:28.707838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:28.707838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:28.707838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:28.707838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:28.707838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:28.707838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:28.707838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:28.707838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:28.708102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:28.708102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:28.708102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:28.708102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:28.708102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:28.708102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:28.708102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:28.708102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:28.708102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:28.708102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:28.708102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:28.708102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:28.708102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:28.708402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:28.708402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:28.708727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:28.708727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:28.708727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:28.708727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:28.708727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:28.708727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:28.708727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:28.708727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:28.708727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:28.708727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:28.708727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:28.708727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:28.708727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:28.708727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:28.708727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:28.708727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:28.709089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:28.709089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:28.709089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:28.709089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:28.709089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:28.709089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:28.709089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:28.709089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:28.709089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:28.709089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:28.709089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:28.709089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:28.709089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:28.709089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:28.709089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:28.709089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:28.709089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:28.709089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:28.709089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:28.709449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:28.709449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:28.709449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:30.746929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:30.748863 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:30.748939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:30.752447 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:30.753373 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:30.753373 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:30.753373 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:30.753373 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:30.753373 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:30.753558 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:30.753558 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:30.753673 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0ac50238-6771-42e2-8e62-db4ee957f87e, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:30.753673 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:30.753804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.755887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:30.755887 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:30.755887 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:30.755887 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:30.755887 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:30.755887 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:30.755887 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.755887 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:30.760049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.760049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:30.760049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:30.760049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:30.760049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.760049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:30.760049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:30.760049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.760049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:30.760049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:30.760049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:30.760049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.760323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:30.760323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:30.760323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:30.760323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:30.760323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:30.760323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:30.760323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:30.760323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:30.760323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:30.760323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:30.760323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:30.760323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:30.760323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:30.760549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.760549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:30.760549 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:30.760906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.760906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:30.760906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:30.760906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:30.760906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:30.760906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:30.760906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.760906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:30.760906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:30.760906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.760906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:30.760906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:30.760906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.760906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:30.760906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:30.760906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.761189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:30.761189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:30.761189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.761189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:30.761189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:30.761189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:30.761189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.761189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:30.761189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:30.761189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:30.761189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:30.761189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:30.761189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:30.761189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:30.761189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:30.761189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:30.761189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:30.761189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:30.761189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:30.761508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:30.761508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.761508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:30.761508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:30.761508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:30.761508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:30:30.761617 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:30.761725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.762177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:30.762177 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:30.762177 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:30.762177 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:30.762177 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:30.762177 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:30.762177 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.762177 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:30.762681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.762681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:30.762681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:30.762681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:30.762681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.762681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:30.762681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:30.762681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:30.762681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.762681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:30.762681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:30.762681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:30.762681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:30.762681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:30.762681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:30.762938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:30.762938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:30.762938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:30.762938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:30.762938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:30.762938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:30.762938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:30.762938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.762938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:30.763212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.763212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:30.763212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:30.763212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:30.763212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.763212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:30.763212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:30.763212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:30:30.763212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:30.763212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:30.763212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.763212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:30.763447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:30.763447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:30.763447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.763447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:30.763447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:30.763447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:30.763447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:30.763447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:30.763447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:30.763447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:30.763447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:30.763447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:30.763447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:30.763447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:30.763655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:30.763655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:30.763655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.763655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:30.764390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.764390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:30.764390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:30.764390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:30.764390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:30.764390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:30.764390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.764390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:30.764390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:30.764390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.764390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:30.764390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:30.764390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.764390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:30.764390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:30.764390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.764690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:30.764690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:30.764690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:30:30.764690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:30.764690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:30.764690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.764690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:30.764690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:30.764690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:30.764690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.764690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:30.764690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:30.764690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:30.764690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:30.764690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:30.764690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:30.764690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:30.764690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:30.764968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:30.764968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:30.764968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:30.764968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:30.764968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:30.764968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.764968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:30.773493 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:30.776527 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:30.776618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:30.776858 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:30.778872 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:30.778872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.779218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:30.779218 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:30.779218 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:30.779218 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:30.779218 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:30.779218 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:30.779218 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.779218 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:30.780179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.780179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:30.780179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:30.780179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:30.780179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.780179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:30.780179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:30.780179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:30.780179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.780179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:30.780179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:30.780179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:30.780179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:30.780179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:30.780179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:30.780606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:30.780606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:30.780606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:30.780606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:30.780606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:30.780606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:30.780606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:30.780606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.780606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:30.780606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.780606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:30.780606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:30.780606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:30.780606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.780975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:30.780975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:30.780975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:30.780975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:30.780975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.780975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:30.780975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:30.780975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:30.780975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.780975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:30.780975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:30.780975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:30.780975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:30.781426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:30.781426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:30.781426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:30.781426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:30.781426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:30.781426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:30.781426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:30.781426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:30.781426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:30.781426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.781426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:30.782432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.782432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:30.782432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:30.782432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:30.782432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:30.782432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:30.782432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.782432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:30.782432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:30.782432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.782432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:30.782432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:30.782432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.782432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:30.782432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:30.782432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.782432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:30.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:30.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:30.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:30.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:30.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:30.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:30.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:30.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:30.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:30.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:30.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:30.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:30.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:30.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:30.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:30.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:30.782934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:30.783916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:30.783916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:30.783916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.783916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:30.783916 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:30.783916 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:30.783916 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:30.783916 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:30.784241 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:30.784698 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:30.784880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.785303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:30.785303 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:30.785303 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:30.785303 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:30.785303 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:30.785303 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:30.785303 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.785303 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:30.785898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.785898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:30.785898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:30.785898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:30.785898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.785898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:30.785898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:30.785898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.785898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:30.785898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:30.785898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:30.785898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.786319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:30.786319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:30.786319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:30.786319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:30.786319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:30.786319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:30.786319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:30.786319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:30.786319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:30.786319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:30.786319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:30.786319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:30.786319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:30.786725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.786725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:30.786908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.786908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:30.786908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:30.786908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:30.786908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:30.786908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:30.786908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.786908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:30.786908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:30.786908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.786908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:30.786908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:30.786908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.786908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:30.786908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:30.786908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:30.787476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:30.787476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:30.787476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.787476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:30.787476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:30.787476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:30.787476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:30.787476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:30.787476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:30.787476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:30.787476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:30.787476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:30.787476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:30.787476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:30.787476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:30.787476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:30.787476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:30.787476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:30.787476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:30.788054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:30.788054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:30.788054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:33.674620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:33.675103 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:33.675103 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:33.675841 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:33.711196 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:33.711696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:33.712439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:33.712439 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:33.712439 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:33.712439 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:33.712439 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:33.712439 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:33.712439 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:33.712439 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:33.713137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:33.713137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:33.713137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:33.713137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:33.713137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:33.713137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:33.713137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:33.713137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:33.713137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:33.713137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:33.713137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:33.713137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:33.713382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:33.713382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:33.713382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:33.713382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:33.713382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:33.713382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:33.713382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:33.713382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:33.713382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:33.713382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:33.713382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:33.713382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:33.713382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:33.713653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:33.713653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:33.714071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:33.714071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:33.714071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:33.714071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:33.714071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:33.714071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:33.714071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:33.714071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:33.714071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:33.714071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:33.714071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:33.714071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:33.714071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:33.714071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:33.714071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:33.714071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:33.714387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:33.714387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:33.714387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:33.714387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:33.714387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:33.714387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:33.714387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:33.714387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:33.714387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:33.714387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:33.714387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:33.714387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:33.714387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:33.714387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:33.714387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:33.714387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:33.714387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:33.714387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:33.714387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:33.714717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:33.714717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:33.714717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:35.749162 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:35.749750 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:35.749750 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:35.750905 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:35.751202 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ac86f889-a3b6-4c1a-bb1a-34eba1b837c9, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:35.751406 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:35.751538 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:35.751652 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:35.751758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:35.751896 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:35.752016 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:35.752121 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f7c5fe6-09b5-4a3f-9371-88eb30470906, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:35.752224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.752793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:35.752793 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:35.752793 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:35.752793 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:35.752793 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:35.752793 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:35.752793 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.752793 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:35.754656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.754656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:35.754656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:35.754656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:35.754656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.754656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:35.754656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:35.754656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.754656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:35.754656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:35.754656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:35.754656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.755050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:35.755050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:35.755050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:35.755050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:35.755050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:35.755050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:35.755050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:35.755050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:35.755050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:35.755050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:35.755050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:35.755050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:35.755050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:35.755507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.755507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:35.755507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.755507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:35.755507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:35.755507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:35.755507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:35.755507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:35.755507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.755507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:35.755507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:35.755507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.755507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:35.755507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:35.755507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.755891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:35.755891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:35.755891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.755891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:35.755891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:35.755891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.755891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:35.755891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:35.755891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:35.755891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.755891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:35.755891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:35.755891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:35.755891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:35.755891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:35.755891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:35.755891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:35.755891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:35.755891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:35.756311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:35.756311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:35.756311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:35.756311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:35.756311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.756311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:35.761214 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:35.763341 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:35.763341 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:35.763341 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:30:35.764227 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:35.764530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.765069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:35.765069 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:35.765069 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:35.765069 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:35.765069 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:35.765069 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:35.765069 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.765069 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:35.765832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.765832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:35.765832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:35.765832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:35.765832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.765832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:35.765832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:35.765832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:35.765832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.765832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:35.765832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:35.765832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:35.765832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:35.765832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:35.765832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:35.766201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:35.766201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:35.766201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:35.766201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:35.766201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:35.766201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:35.766201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:35.766201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.766201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:35.766554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.766554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:35.766554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:35.766554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:35.766554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.766554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:35.766554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:35.766554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:30:35.766554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:35.766554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:35.766554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.766554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:35.766777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:35.766777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:35.766777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.766777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:35.766777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:35.766777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:35.766777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:35.766777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:35.766777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:35.766777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:35.766777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:35.766777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:35.766777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:35.766777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:35.767024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:35.767024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:35.767024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.767024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:35.768900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.768900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:35.768900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:35.768900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:35.768900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:35.768900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:35.768900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.768900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:35.768900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:35.768900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.768900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:35.768900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:35.768900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.768900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:35.768900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:35.768900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.769302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:35.769302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:35.769302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:30:35.769302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:35.769302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:35.769302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.769302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:35.769302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:35.769302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:35.769302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.769302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:35.769302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:35.769302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:35.769302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:35.769302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:35.769302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:35.769302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:35.769302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:35.769578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:35.769578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:35.769578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:35.769578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:35.769578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:35.769578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.769578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:35.778974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:35.778974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:35.778974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:35.778974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:35.779181 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:35.779367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.780111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:35.780111 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:35.780111 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:35.780111 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:35.780111 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:35.780111 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:35.780111 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.780111 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:35.781359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.781359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:35.781359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:35.781359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:35.781359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.781359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:35.781359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:35.781359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:35.781359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.781359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:35.781359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:35.781359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:35.781359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:35.781359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:35.781359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:35.781676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:35.781676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:35.781676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:35.781676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:35.781676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:35.781676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:35.781676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:35.781676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.781676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:35.781676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.781676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:35.781676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:35.781676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:35.781676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.781920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:35.781920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:35.781920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:35.781920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:35.781920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.781920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:35.781920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:35.781920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:35.781920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.781920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:35.781920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:35.781920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:35.781920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:35.782198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:35.782198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:35.782198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:35.782198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:35.782198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:35.782198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:35.782198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:35.782198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:35.782198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:35.782198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.782198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:35.782198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.782546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:35.782546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:35.782546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:35.782546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:35.782546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:35.782546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.782546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:35.782546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:35.782546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.782546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:35.782546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:35.782546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.782546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:35.782546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:35.782546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.782546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:35.782546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:35.782546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:35.782546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.782921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:35.783633 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:35.783835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:35.783938 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:35.784100 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:35.784544 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:35.784892 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:35.785181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.785702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:35.785702 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:35.785702 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:35.785702 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:35.785702 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:35.785702 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:35.785702 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.785702 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:35.786688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.786688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:35.786688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:35.786688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:35.786688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.786688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:35.786688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:35.786688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.786688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:35.786688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:35.786688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:35.786688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.786948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:35.786948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:35.786948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:35.786948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:35.786948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:35.786948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:35.786948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:35.786948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:35.786948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:35.786948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:35.786948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:35.786948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:35.786948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:35.787196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.787196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:35.791334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.791334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:35.791334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:35.791334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:35.791334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:35.791334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:35.791334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.791334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:35.791334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:35.791334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.791334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:35.791334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:35.791334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.791334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:35.791334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:35.791334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:35.791730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:35.791730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:35.791730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.791730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:35.791730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:35.791730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:35.791730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:35.791730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:35.791730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:35.791730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:35.791730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:35.791730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:35.791730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:35.791730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:35.791730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:35.791730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:35.791730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:35.791730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:35.791730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:35.792051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:35.792051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:35.792051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:38.674769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:38.675199 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:38.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:38.676616 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:38.730855 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:38.731073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:38.732745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:38.732745 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:38.732745 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:38.732745 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:38.732745 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:38.732745 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:38.732745 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:38.732745 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:38.733486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.057 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:38.733486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:38.733486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:38.733486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:38.733486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:38.733486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:38.733486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:38.733486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:38.733486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:38.733486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:38.733486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:38.733486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:38.733800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:38.733800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:38.733800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:38.733800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:38.733800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:38.733800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:38.733800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:38.733800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:38.733800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:38.733800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:38.733800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:38.733800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:38.733800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:38.734146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:38.734146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:38.734238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:38.734238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:38.734238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:38.734238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:38.734238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:38.734238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:38.734238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:38.734238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:38.734238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:38.734238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:38.734238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:38.734238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:38.734238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:38.734238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:38.734238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:38.734238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:38.734659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:38.734659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:38.734659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:38.734659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:38.734659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:38.734659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:38.734659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:38.734659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:38.734659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:38.734659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:38.734659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:38.734659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:38.734659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:38.734659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:38.734659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:38.734659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:38.734659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:38.734659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:38.734659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:38.735191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:38.735191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:38.735191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:40.751497 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:40.753212 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:40.753730 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:40.757472 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:40.757868 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:40.758153 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:40.761359 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:40.761582 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:40.761896 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:40.762009 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:40.762211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.762696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:40.763390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:40.763390 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:40.763390 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:40.763390 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:40.763390 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:40.763390 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:40.763390 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.763390 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:40.764208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.764208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:40.764208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:40.764208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:40.764208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.764208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:40.764208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:40.764208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.764208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:40.764208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:40.764208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:40.764208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.764426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:40.764426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:40.764426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:40.764426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:40.764426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:40.764426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:40.764426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:40.764426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:40.764426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:40.764426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:40.764426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:40.764426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:40.764426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:40.764632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.764632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:40.765046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.765046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:40.765046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:40.765046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:40.765046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:40.765046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:40.765046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.765046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:40.765046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:40.765046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.765046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:40.765046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:40.765046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.765046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:40.765046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:40.765046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.765327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:40.765327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:40.765327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.765327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:40.765327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:40.765327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:40.765327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.765327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:40.765327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:40.765327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:40.765327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:40.765327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:40.765327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:40.765327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:40.765327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:40.765327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:40.765327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:40.765327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:40.765327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:40.765697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:40.765697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.765697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:40.765697 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:40.765697 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:40.765697 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:30:40.766130 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:40.766488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.767085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:40.767085 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:40.767085 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:40.767085 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:40.767085 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:40.767085 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:40.767085 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.767085 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:40.767714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.767714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:40.767714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:40.767714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:40.767714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.767714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:40.767714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:40.767714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:40.767714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.767714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:40.767714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:40.767714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:40.767714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:40.767714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:40.767714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:40.767963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:40.767963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:40.767963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:40.767963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:40.767963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:40.767963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:40.767963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:40.767963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.767963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:40.768322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.768322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:40.768322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:40.768322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:40.768322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.768322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:40.768322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:40.768322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:30:40.768322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:40.768322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:40.768322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.768322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:40.768547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:40.768547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:40.768547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.768547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:40.768547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:40.768547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:40.768547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:40.768547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:40.768547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:40.768547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:40.768547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:40.768547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:40.768547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:40.768547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:40.768775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:40.768775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:40.768775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.768775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:40.769069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.769069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:40.769069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:40.769069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:40.769069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:40.769069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:40.769069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.769069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:40.769069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:40.769069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.769069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:40.769069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:40.769069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.769069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:40.769069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:40.769069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.769335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:40.769335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:40.769335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:30:40.769335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:40.769335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:40.769335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.769335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:40.769335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:40.769335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:40.769335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.769335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:40.769335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:40.769335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:40.769335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:40.769335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:40.769335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:40.769335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:40.769335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:40.769654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:40.769654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:40.769654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:40.769654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:40.769654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:40.769654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.769654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:40.778531 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:40.778999 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:40.779113 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:40.779309 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:40.779867 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:40.780166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.780808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:40.780808 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:40.780808 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:40.780808 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:40.780808 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:40.780808 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:40.780808 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.780808 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:40.781766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.781766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:40.781766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:40.781766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:40.781766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.781766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:40.781766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:40.781766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:40.781766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.781766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:40.781766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:40.781766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:40.781766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:40.781766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:40.781766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:40.782037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:40.782037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:40.782037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:40.782037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:40.782037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:40.782037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:40.782037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:40.782037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.782037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:40.782698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.782698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:40.782698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:40.782698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:40.782698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.782698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:40.782698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:40.782698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:40.782698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:40.782698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.782698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:40.782924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:40.782924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:40.782924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.782924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:40.782924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:40.782924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:40.782924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:40.782924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:40.782924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:40.782924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:40.782924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:40.782924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:40.782924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:40.782924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:40.783157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:40.783157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:40.783157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.783157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:40.783910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.783910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:40.783910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:40.783910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:40.783910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:40.783910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:40.783910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.783910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:40.783910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:40.783910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.783910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:40.783910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:40.783910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.783910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:40.783910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:40.783910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.783910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:40.784209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:40.784209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:40.784209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:40.784209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.784209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:40.784209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:40.784209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:40.784209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.784209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:40.784209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:40.784209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:40.784209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:40.784209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:40.784209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:40.784209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:40.784209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:40.784209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:40.784209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:40.784209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:40.784550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:40.784550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:40.784550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.784550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:40.784550 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:40.784670 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:40.784790 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:40.784975 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:40.785442 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:40.785775 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:40.786124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.786772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:40.786772 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:40.786772 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:40.786772 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:40.786772 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:40.786772 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:40.786772 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.786772 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:40.787747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.787747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:40.787747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:40.787747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:40.787747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.787747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:40.787747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:40.787747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.787747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:40.787747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:40.787747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:40.787747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.787976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:40.787976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:40.787976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:40.787976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:40.787976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:40.787976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:40.787976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:40.787976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:40.787976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:40.787976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:40.787976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:40.787976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:40.787976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:40.788211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.788211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:40.788851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.788851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:40.788851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:40.788851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:40.788851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:40.788851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:40.788851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.788851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:40.788851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:40.788851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.788851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:40.788851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:40.788851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.788851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:40.788851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:40.788851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:40.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:40.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:40.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:40.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:40.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:40.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:40.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:40.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:40.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:40.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:40.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:40.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:40.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:40.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:40.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:40.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:40.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:40.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:40.789445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:40.789445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:40.789445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:43.675645 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:43.676726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:43.676831 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:43.677521 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:43.711820 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:43.712095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:43.712589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:43.712589 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:43.712589 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:43.712589 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:43.712589 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:43.712589 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:43.712589 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:43.712589 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:43.713310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:43.713310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:43.713310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:43.713310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:43.713310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:43.713310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:43.713310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:43.713310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:43.713310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:43.713310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:43.713310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:43.713310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:43.713560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:43.713560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:43.713560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:43.713560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:43.713560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:43.713560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:43.713560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:43.713560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:43.713560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:43.713560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:43.713560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:43.713560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:43.713560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:43.713754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:43.713754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:43.714011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:43.714011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:43.714011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:43.714011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:43.714011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:43.714011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:43.714011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:43.714011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:43.714011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:43.714011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:43.714011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:43.714011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:43.714011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:43.714011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:43.714011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:43.714011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:43.714288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:43.714288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:43.714288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:43.714288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:43.714288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:43.714288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:43.714288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:43.714288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:43.714288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:43.714288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:43.714288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:43.714288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:43.714288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:43.714288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:43.714288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:43.714288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:43.714288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:43.714288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:43.714288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:43.714600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:43.714600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:43.714600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:45.752464 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:45.752953 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:45.752953 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:45.753854 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:45.754094 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:45.754279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:45.754455 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:45.754720 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:45.754871 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:45.755040 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:45.755168 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:45.755321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.756217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:45.756217 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:45.756217 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:45.756217 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:45.756217 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:45.756217 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:45.756217 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.756217 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:45.757155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.757155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:45.757155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:45.757155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:45.757155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.757155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:45.757155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:45.757155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.757155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:45.757155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:45.757155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:45.757155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.757584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:45.757584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:45.757584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:45.757584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:45.757584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:45.757584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:45.757584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:45.757584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:45.757584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:45.757584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:45.757584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:45.757584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:45.757584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:45.757965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.757965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:45.758317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.758317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:45.758317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:45.758317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:45.758317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:45.758317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:45.758317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.758317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:45.758317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:45.758317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.758317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:45.758317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:45.758317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.758317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:45.758317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:45.758317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.758710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:45.758710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:45.758710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.758710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:45.758710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:45.758710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:45.758710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.758710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:45.758710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:45.758710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:45.758710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:45.758710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:45.758710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:45.758710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:45.758710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:45.758710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:45.758710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:45.758710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:45.758710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:45.759226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:45.759226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.759226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:45.765411 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:45.767217 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:45.767634 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:45.768037 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:30:45.771083 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:45.771344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.772062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:45.772062 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:45.772062 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:45.772062 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:45.772062 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:45.772062 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:45.772062 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.772062 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:45.772937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.772937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:45.772937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:45.772937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:45.772937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.772937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:45.772937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:45.772937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:45.772937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.772937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:45.772937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:45.772937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:45.772937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:45.772937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:45.772937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:45.773445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:45.773445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:45.773445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:45.773445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:45.773445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:45.773445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:45.773445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:45.773445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.773445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:45.776697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.776697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:45.776697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:45.776697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:45.776697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.776697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:45.776697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:45.776697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:30:45.776697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:45.776697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:45.776697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.776697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:45.777095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:45.777095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:45.777095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.777095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:45.777095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:45.777095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:45.777095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:45.777095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:45.777095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:45.777095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:45.777095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:45.777095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:45.777095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:45.777095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:45.777470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:45.777470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:45.777470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.777470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:45.777878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.777878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:45.777878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:45.777878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:45.777878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:45.777878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:45.777878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.777878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:45.777878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:45.777878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.777878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:45.777878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:45.777878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.777878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:45.777878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:45.777878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.778357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:45.778357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:45.778357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:30:45.778357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:45.778357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:45.778357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.778357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:45.778357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:45.778357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:45.778357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.778357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:45.778357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:45.778357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:45.778357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:45.778357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:45.778357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:45.778357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:45.778357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:45.778786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:45.778786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:45.778786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:45.778786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:45.778786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:45.778786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.778786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:45.778968 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:45.779144 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:45.779237 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:45.779446 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:45.779886 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:45.780650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.782851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:45.782851 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:45.782851 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:45.782851 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:45.782851 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:45.782851 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:45.782851 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.782851 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:45.783603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.783603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:45.783603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:45.783603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:45.783603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.783603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:45.783603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:45.783603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:45.783603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.783603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:45.783603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:45.783603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:45.783603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:45.783603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:45.783603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:45.784025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:45.784025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:45.784025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:45.784025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:45.784025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:45.784025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:45.784025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:45.784025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.784025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:45.784655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.784655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:45.784655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:45.784655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:45.784655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.784655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:45.784655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:45.784655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:45.784655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:45.784655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.784655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:45.784976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:45.784976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:45.784976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.784976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:45.784976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:45.784976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:45.784976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:45.784976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:45.784976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:45.784976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:45.784976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:45.784976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:45.784976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:45.784976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:45.785374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:45.785374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:45.785374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.785374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:45.785374 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:45.786083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.786083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:45.786083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:45.786083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:45.786083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:45.786083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:45.786083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.786083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:45.786083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:45.786083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.786083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:45.786083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:45.786083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.786083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:45.786083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:45.786083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.786083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:45.786587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:45.786587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:45.786587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:45.786587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.786587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:45.786587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:45.786587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:45.786587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.786587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:45.786587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:45.786587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:45.786587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:45.786587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:45.786587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:45.786587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:45.786587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:45.786587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:45.786587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:45.786587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:45.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:45.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:45.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:45.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:45.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:45.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:45.787047 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:45.787312 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:45.787502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.787946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:45.787946 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:45.787946 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:45.787946 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:45.787946 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:45.787946 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:45.787946 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.787946 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:45.788767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.788767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:45.788767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:45.788767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:45.788767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.788767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:45.788767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:45.788767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.788767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:45.788767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:45.788767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:45.788767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.789236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:45.789236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:45.789236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:45.789236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:45.789236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:45.789236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:45.789236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:45.789236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:45.789236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:45.789236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:45.789236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:45.789236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:45.789236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:45.789617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.789617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:45.790351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.790351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:45.790351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:45.790351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:45.790351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:45.790351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:45.790351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.790351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:45.790351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:45.790351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.790351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:45.790351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:45.790351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.790351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:45.790351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:45.790351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:45.790829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:45.790829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:45.790829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.790829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:45.790829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:45.790829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:45.790829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:45.790829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:45.790829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:45.790829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:45.790829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:45.790829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:45.790829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:45.790829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:45.790829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:45.790829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:45.790829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:45.790829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:45.790829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:45.791378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:45.791378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:45.791378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:48.677019 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:48.677431 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:48.677565 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:48.678270 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:48.711397 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:48.711624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:48.713541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:48.713541 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:48.713541 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:48.713541 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:48.713541 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:48.713541 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:48.713541 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:48.713541 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:48.714274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:48.714274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:48.714274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:48.714274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:48.714274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:48.714274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:48.714274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:48.714274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:48.714274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:48.714274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:48.714274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:48.714274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:48.714630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:48.714630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:48.714630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:48.714630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:48.714630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:48.714630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:48.714630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:48.714630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:48.714630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:48.714630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:48.714630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:48.714630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:48.714630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:48.714875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:48.714875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:48.715041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:48.715041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:48.715041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:48.715041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:48.715041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:48.715041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:48.715041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:48.715041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:48.715041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:48.715041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:48.715041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:48.715041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:48.715041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:48.715041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:48.715041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:48.715041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:48.715352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:48.715352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:48.715352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:48.715352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:48.715352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:48.715352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:48.715352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:48.715352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:48.715352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:48.715352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:48.715352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:48.715352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:48.715352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:48.715352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:48.715352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:48.715352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:48.715352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:48.715352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:48.715352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:48.715817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:48.715817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:48.715817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:50.753518 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:50.753911 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:50.754003 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:50.755635 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:50.755811 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:50.755934 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:50.756320 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:50.756429 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:50.756603 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:50.756723 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:50.756824 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:50.756918 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:50.757068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.757568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:50.757568 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:50.757568 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:50.757568 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:50.757568 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:50.757568 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:50.757568 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.757568 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:50.758554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.758554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:50.758554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:50.758554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:50.758554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.758554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:50.758554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:50.758554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.758554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:50.758554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:50.758554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:50.758554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.758778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:50.758778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:50.758778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:50.758778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:50.758778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:50.758778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:50.758778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:50.758778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:50.758778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:50.758778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:50.758778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:50.758778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:50.758778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:50.759001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.759001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:50.759699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.759699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:50.759699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:50.759699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:50.759699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:50.759699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:50.759699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.759699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:50.759699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:50.759699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.759699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:50.759699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:50.759699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.759699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:50.759699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:50.759699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.759972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:50.759972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:50.759972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.759972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:50.759972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:50.759972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:50.759972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.759972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:50.759972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:50.759972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:50.759972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:50.759972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:50.759972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:50.759972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:50.759972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:50.759972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:50.759972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:50.759972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:50.759972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:50.760296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:50.760296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.760296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:50.767443 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:50.769276 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:50.769276 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:50.769276 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:30:50.769276 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:50.769276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.769276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:50.769276 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:50.769276 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:50.769276 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:50.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:50.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:50.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:50.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:50.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:50.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:50.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:50.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:50.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:50.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:50.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:50.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:50.770451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:50.770451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:50.770451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:50.770451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:50.770451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:50.770451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:50.770451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:50.770451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:50.770451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:50.770451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:50.770451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.770451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:50.770451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.770451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:50.770793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:50.770793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:50.770793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.770793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:50.770793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:50.770793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:30:50.770793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:50.770793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:50.770793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.770793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:50.770793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:50.770793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:50.770793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.771159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:50.771159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:50.771159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:50.771159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:50.771159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:50.771159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:50.771159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:50.771159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:50.771159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:50.771159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:50.771159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:50.771159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:50.771159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:50.771516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.771516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:50.771516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.771516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:50.771516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:50.771516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:50.771516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:50.771516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:50.771516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.771516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:50.771516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:50.771516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.771516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:50.771516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:50.771516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.771954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:50.771954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:50.771954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.771954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:50.771954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:50.771954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:30:50.771954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:50.771954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:50.771954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.771954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:50.771954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:50.771954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:50.771954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.771954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:50.771954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:50.771954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:50.771954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:50.771954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:50.771954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:50.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:50.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:50.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:50.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:50.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:50.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:50.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:50.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.774762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:50.779189 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:50.779631 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:50.779631 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:50.779756 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:50.780042 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:50.780222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.780706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:50.780706 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:50.780706 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:50.780706 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:50.780706 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:50.780706 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:50.780706 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.780706 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:50.781244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.781244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:50.781244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:50.781244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:50.781244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.781244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:50.781244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:50.781244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:50.781244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.781244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:50.781244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:50.781244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:50.781244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:50.781244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:50.781244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:50.781529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:50.781529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:50.781529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:50.781529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:50.781529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:50.781529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:50.781529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:50.781529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.781529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:50.782099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.782099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:50.782099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:50.782099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:50.782099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.782099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:50.782099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:50.782099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:50.782099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:50.782099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.782099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:50.782846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:50.782846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:50.782846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.782846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:50.782846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:50.782846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:50.782846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:50.782846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:50.782846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:50.782846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:50.782846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:50.782846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:50.782846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:50.782846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:50.783067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:50.783067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:50.783067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.783067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:50.783067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.783067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:50.783067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:50.783067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:50.783067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:50.783067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:50.783067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.783067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:50.783067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:50.783067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.783316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:50.783316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:50.783316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.783316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:50.783316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:50.783316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.783316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:50.783316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:50.783316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:50.783316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:50.783316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.783316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:50.783316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:50.783316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:50.783316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.783316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:50.783316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:50.783316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:50.783316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:50.783620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:50.783620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:50.783620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:50.783620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:50.783620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:50.783620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:50.783620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:50.783620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:50.783620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:50.783620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.783620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:50.786333 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:50.786520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:50.786610 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:50.787267 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:50.787267 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:50.787267 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:50.787478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.788657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:50.788657 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:50.788657 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:50.788657 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:50.788657 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:50.788657 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:50.788657 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.788657 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:50.789339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.789339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:50.789339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:50.789339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:50.789339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.789339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:50.789339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:50.789339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.789339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:50.789339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:50.789339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:50.789339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.789559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:50.789559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:50.789559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:50.789559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:50.789559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:50.789559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:50.789559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:50.789559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:50.789559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:50.789559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:50.789559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:50.789559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:50.789559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:50.789767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.789767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:50.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:50.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:50.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:50.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:50.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:50.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:50.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:50.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:50.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:50.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:50.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:50.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:50.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:50.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:50.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:50.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:50.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:50.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:50.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:50.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:50.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:50.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:50.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:50.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:50.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:50.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:50.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:50.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:50.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:50.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:50.792321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:50.792321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:50.792321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:53.679159 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:53.679159 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:53.679159 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:53.679159 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:53.691906 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=240 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:30:53.729279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:53.729279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:53.729279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:53.729279 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:53.729279 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:53.729279 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:53.729279 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:53.729279 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:53.729279 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:53.729279 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:53.729279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.050 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:53.729279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:53.729279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:53.729577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:53.729577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:53.729577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:53.729577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:53.729577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:53.729577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:53.729577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:53.729577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:53.729577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:53.729577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:53.729577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:53.729577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:53.729577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:53.729577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:53.729804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:53.729804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:53.729804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:53.729804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:53.729804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:53.729804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:53.729804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:53.729804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:53.729804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:53.729804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:53.729804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:53.729804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:53.729999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:53.729999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:53.729999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:53.729999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:53.729999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:53.729999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:53.729999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:53.729999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:53.729999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:53.729999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:53.729999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:53.729999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:53.729999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:53.729999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:53.729999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:53.729999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:53.729999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:53.729999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:53.729999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:53.730300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:53.730300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:53.730300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:53.730300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:53.730300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:53.730300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:53.730300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:53.730300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:53.730300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:53.730300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:53.730300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:53.730300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:53.730300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:53.730300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:53.730300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:53.730300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:53.730300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:55.753933 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:55.754695 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:55.754777 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:55.755815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:55.756014 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:55.756134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:55.756481 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:55.756481 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:55.756600 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:55.756600 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:55.756698 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:55.756806 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:55.756901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.758674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:55.758674 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:55.758674 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:55.758674 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:55.758674 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:55.758674 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:55.758674 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.758674 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:55.759375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.759375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:55.759375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:55.759375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:55.759375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.759375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:55.759375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:55.759375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.759375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:55.759375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:55.759375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:55.759375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.759825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:55.759825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:55.759825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:55.759825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:55.759825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:55.759825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:55.759825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:55.759825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:55.759825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:55.759825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:55.759825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:55.759825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:55.759825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:55.760417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.760417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:55.760417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.760417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:55.760417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:55.760417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:55.760417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:55.760417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:55.760417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.760417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:55.760417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:55.760417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.760417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:55.760417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:55.760417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.761212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:55.761212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:55.761212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.761212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:30:55.761212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:55.761212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.761212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:55.761212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:55.761212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:55.761212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.761212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:55.761212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:55.761212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:55.761212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:55.761212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:55.761212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:55.761212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:55.761212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:55.761212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:55.761733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:55.761733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:55.761733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:55.761733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:55.761733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.761733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:55.777386 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:55.777717 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:55.778299 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:55.778299 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:30:55.778299 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:55.778537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.779152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:55.779152 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:55.779152 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:55.779152 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:55.779152 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:55.779152 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:55.779152 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.779152 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:55.779781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.779781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:55.779781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:55.779781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:55.779781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.779781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:55.779781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:55.779781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:55.779781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.779781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:55.779781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:55.779781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:55.779781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:55.779781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:55.779781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:55.780439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:55.780439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:55.780439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:55.780439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:55.780439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:55.780439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:55.780439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:55.780439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.780439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:55.780927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.780927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:55.780927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:55.780927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:55.780927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.780927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:55.780927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:55.780927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:30:55.780927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:55.780927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:55.780927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.780927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:55.781491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:55.781491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:55.781491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.781491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:55.781491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:55.781491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:55.781491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:55.781491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:55.781491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:55.781491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:55.781491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:55.781491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:55.781491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:55.781491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:55.781977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:55.781977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:55.781977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.781977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:55.781977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.781977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:55.781977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:55.781977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:55.781977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:55.781977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:55.781977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.781977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:55.781977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:55.781977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.782678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:55.782678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:55.782678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.782678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:55.782678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:55.782678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.782678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:30:55.782678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:30:55.782678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:30:55.782678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:55.782678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:55.782678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.782678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:55.782678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:55.782678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:55.782678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.782678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:55.782678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:55.782678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:55.783598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:55.783598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:55.783598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:55.783598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:55.783598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:55.783598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:55.783598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:55.783598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:55.783598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:55.783598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:55.783598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.783598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:55.783598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:55.783598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:55.784339 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:55.784339 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:55.784339 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:55.784339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.784339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:55.784339 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:55.784339 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:55.784339 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:55.784339 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:55.784339 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:55.784339 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.784339 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:55.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:30:55.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:55.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:55.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:55.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:55.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:30:55.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:30:55.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:55.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:30:55.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:55.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:30:55.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:30:55.785451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:55.785451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:30:55.785451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:55.785451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:30:55.785451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:30:55.785451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:55.785451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:30:55.785451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.785451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:30:55.785451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.785451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:55.785451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:55.785451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:55.785451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.786074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:55.786074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:55.786074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:55.786074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:55.786074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.786074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:55.786074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:55.786074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:55.786074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.786074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:55.786074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:55.786074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:55.786074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:55.786879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:55.786879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:55.786879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:55.786879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:55.786879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:55.786879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:55.786879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:55.786879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:55.786879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:55.786879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.786879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:55.786879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.787214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:55.787214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:55.787214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:55.787214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:55.787214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:55.787214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.787214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:55.787214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:55.787214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.787214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:55.787214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:55.787214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.787214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:55.787214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:55.787214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.787214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:30:55.787214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:30:55.787214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:30:55.787214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.788246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:55.788878 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:55.788878 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:55.788878 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:55.788878 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:55.788878 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:55.789602 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:55.789602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.789866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:55.789866 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:55.789866 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:55.789866 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:55.789866 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:55.789866 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:55.789866 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.789866 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:55.790450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.790450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:55.790450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:55.790450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:55.790450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.790450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:55.790450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:55.790450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.790450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:55.790450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:55.790450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:55.790450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.790772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:55.790772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:55.790772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:55.790772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:55.790772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:55.790772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:55.790772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:55.790772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:55.790772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:55.790772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:55.790772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:55.790772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:55.790772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:55.791143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.791143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:55.791143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.791143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:55.791143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:55.791143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:55.791143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:55.791143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:55.791143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.791143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:55.791143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:55.791143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.791143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:55.791143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:55.791143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.792719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:55.792719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:55.792719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:55.792719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:30:55.792719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:55.792719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.792719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:55.792719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:55.792719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:55.792719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:55.792719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:55.792719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:55.792719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:55.792719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:55.792719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:55.792719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:55.792719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:55.792719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:55.792719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:55.793801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:55.793801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:55.793801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:55.793801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:55.793801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:55.793801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:58.677906 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:58.678376 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:58.678376 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:58.678848 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:30:58.716378 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:58.717112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:58.717112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:58.717112 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:58.717112 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:58.717112 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:58.717112 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:58.717112 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:58.717112 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:58.717112 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:58.717876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:58.717876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:58.717876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:58.717876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:30:58.717876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:58.717876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:58.717876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:58.717876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:58.717876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:58.717876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:58.717876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:58.717876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:58.718157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:58.718157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:58.718157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:58.718157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:58.718157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:58.718157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:58.718157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:58.718157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:58.718157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:58.718157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:58.718157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:58.718157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:58.718157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:58.718390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:58.718390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:58.719624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:58.719624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:30:58.719624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:30:58.719624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:30:58.719624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:30:58.719624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:30:58.719624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:58.719624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:30:58.719624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:30:58.719624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:58.719624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:30:58.719624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:30:58.719624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:30:58.719624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:30:58.719624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:30:58.719624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:30:58.720091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:30:58.720091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:58.720091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:58.720091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:58.720091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:58.720091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:30:58.720091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:58.720091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:30:58.720091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:58.720091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:30:58.720091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:58.720091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:30:58.720091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:30:58.720091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:58.720091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:30:58.720091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:58.720091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:30:58.720091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:30:58.720091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:58.720435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:30:58.720435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:58.720435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:30:58.857943 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:30:58.859899 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:30:58.860133 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:30:58.868649 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 08 14:30:58.868761 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Number of inconsistencies found at create/update: networks=9, security_groups=5, routers=6, router_ports=6 {{(pid=89274) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 14:30:58.868858 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=20 {{(pid=89274) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 14:30:58.868933 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource fffd4e9d-b343-4a97-8e5f-4015f805fc46 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:58.904745 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, resource={'id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'name': 'public', 'tenant_id': 'ba473288446843d3a29e25267968ba61', 'project_id': 'ba473288446843d3a29e25267968ba61', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['4ee83802-2181-432f-a76d-d3335bbf4e0b', 'b9859e49-8026-492d-a0a4-11326477713e'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-08T14:20:46Z', 'updated_at': '2026-06-08T14:20:52Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:58.904745 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:58.904904 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-0817ce61-1c85-4457-911a-0b8fdccf389d, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:58.904904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:58.904904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:58.904904 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:58.904904 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:58.904904 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:58.904904 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:58.904904 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:58.904904 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:58.904904 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:58.904904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource fffd4e9d-b343-4a97-8e5f-4015f805fc46 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:58.904904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:58.904904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:30:58.905312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:30:58.905312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:30:58.905312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:30:58.905312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:30:58.905312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:58.905312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:58.905312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:58.905312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:58.905312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:58.905312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:58.905312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:58.905312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:58.905312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:58.905695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:58.905695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:58.905695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:58.905695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:58.905695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:58.905695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:58.905695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:58.905695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:58.905695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:58.905695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:58.905695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:58.905695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:58.906067 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 73426422-a7fc-44cd-9fc5-cd2d8fa95c38 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:58.934633 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-73426422-a7fc-44cd-9fc5-cd2d8fa95c38, resource={'id': '73426422-a7fc-44cd-9fc5-cd2d8fa95c38', 'name': 'shared', 'tenant_id': 'ba473288446843d3a29e25267968ba61', 'project_id': 'ba473288446843d3a29e25267968ba61', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['2b53474a-74e5-4c44-bb1b-86c8215bd7ab'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:21:52Z', 'updated_at': '2026-06-08T14:21:52Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 292}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:58.934852 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-73426422-a7fc-44cd-9fc5-cd2d8fa95c38, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:58.935088 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-5c6a587e-b2f4-42c7-9706-8a2e20b0df46, external_ids_update=None, columns={'tag': 292}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:58.935203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:58.936406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:58.936406 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:58.936406 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:58.936406 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:58.936406 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:58.936406 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:58.936406 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:58.936406 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:58.937235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 73426422-a7fc-44cd-9fc5-cd2d8fa95c38 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:58.937235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:58.937235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:30:58.937235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:30:58.937235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:30:58.937235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:30:58.937235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:30:58.937235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:58.937235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:58.937235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:58.937235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:58.937449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:58.937449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:58.937449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:58.937449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:58.937449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:58.937449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:58.937449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:58.937449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:58.937449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:58.937449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:58.937449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:58.937449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:58.937449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:58.937449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:58.937661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:58.937661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:58.937661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:58.937661 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 1d45c7d7-7d44-48e4-8e18-7b3a34b21fff (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:58.965861 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-1d45c7d7-7d44-48e4-8e18-7b3a34b21fff, resource={'id': '1d45c7d7-7d44-48e4-8e18-7b3a34b21fff', 'name': 'tempest-ServerActionsTestOtherA-1380864152-network', 'tenant_id': '2ce3c80c0935406eba76da06872272d9', 'project_id': '2ce3c80c0935406eba76da06872272d9', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['300db623-a76e-47ec-9281-4095dc84f270'], 'standard_attr_id': 574, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:26:46Z', 'updated_at': '2026-06-08T14:26:47Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 19316}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:58.966120 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-1d45c7d7-7d44-48e4-8e18-7b3a34b21fff, col_values=(('external_ids', {'neutron:network_name': 'tempest-ServerActionsTestOtherA-1380864152-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:58.966419 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-9311fe9a-4cb8-427e-b0ba-4aa574961d49, external_ids_update=None, columns={'tag': 19316}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:58.966522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:58.967121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:58.967121 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:58.967121 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:58.967121 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:58.967121 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:58.967121 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:58.967121 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:58.967121 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:58.967942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 1d45c7d7-7d44-48e4-8e18-7b3a34b21fff (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:58.967942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:58.967942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:30:58.967942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:30:58.967942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:30:58.967942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:30:58.967942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:30:58.967942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:58.967942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:58.967942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:58.967942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:58.968145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:58.968145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:58.968145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:58.968145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:58.968145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:58.968145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:58.968145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:58.968145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:58.968145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:58.968145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:58.968145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:58.968145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:58.968145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:58.968145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:58.968421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:58.968421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:58.968421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:58.968421 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource a137566a-eaf1-4f5d-a192-15d63e6c049a (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:59.000712 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-a137566a-eaf1-4f5d-a192-15d63e6c049a, resource={'id': 'a137566a-eaf1-4f5d-a192-15d63e6c049a', 'name': 'private', 'tenant_id': 'ba473288446843d3a29e25267968ba61', 'project_id': 'ba473288446843d3a29e25267968ba61', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['7b3cb50f-b942-4054-b735-5f7be989b8f8', 'ba10f097-6c2b-4188-8d42-e9415eff0039'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:20:42Z', 'updated_at': '2026-06-08T14:20:44Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 26551}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.001061 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-a137566a-eaf1-4f5d-a192-15d63e6c049a, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.001427 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-c4b70b51-9d46-4e6b-a1e7-b3e6a6b8b666, external_ids_update=None, columns={'tag': 26551}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.001549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.002104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:59.002104 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.002104 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:59.002104 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:59.002104 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.002104 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:59.002104 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.002104 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:59.003348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource a137566a-eaf1-4f5d-a192-15d63e6c049a (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.003348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:59.003348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:30:59.003348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:30:59.003348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:30:59.003348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:30:59.003348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:30:59.003348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:59.003348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.003348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.003348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:59.003584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:59.003584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.003584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.003584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:59.003584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:59.003584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:59.003584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:59.003584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:59.003584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:59.003584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:59.003584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.003584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:59.003584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:59.003584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.003890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:59.003890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.003890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:59.003890 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 7615fa0e-1e0e-4d1b-8795-4efc17c573f9 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:59.032383 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-7615fa0e-1e0e-4d1b-8795-4efc17c573f9, resource={'id': '7615fa0e-1e0e-4d1b-8795-4efc17c573f9', 'name': 'tempest-TestShelveInstance-1091677164-network', 'tenant_id': '95d135317c314a42bb44fd26ddacaefd', 'project_id': '95d135317c314a42bb44fd26ddacaefd', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['56c87bff-1908-40f4-94a2-a25b41245bd4'], 'standard_attr_id': 905, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:29:47Z', 'updated_at': '2026-06-08T14:29:47Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 4607}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.032623 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-7615fa0e-1e0e-4d1b-8795-4efc17c573f9, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestShelveInstance-1091677164-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.032909 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-79cb32e9-db84-44e2-84dc-1b931e99ca3c, external_ids_update=None, columns={'tag': 4607}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.033020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.033550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:59.033550 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.033550 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:59.033550 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:59.033550 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.033550 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:59.033550 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.033550 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:59.034706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 7615fa0e-1e0e-4d1b-8795-4efc17c573f9 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.034706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:59.034706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:30:59.034706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:30:59.034706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:30:59.034706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:30:59.034706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:30:59.034706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:59.034706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.034706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.034706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:59.034907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:59.034907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.034907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.034907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:59.034907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:59.034907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:59.034907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:59.034907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:59.034907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:59.034907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:59.034907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.034907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:59.034907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:59.034907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.035117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:59.035117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.035117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:59.035117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 9a4f3019-f88a-411b-a449-099a5c655333 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:59.064950 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-9a4f3019-f88a-411b-a449-099a5c655333, resource={'id': '9a4f3019-f88a-411b-a449-099a5c655333', 'name': 'tempest-TestShelveInstance-458690823-network', 'tenant_id': 'f079905a6a76483682af4fb9dadc4e58', 'project_id': 'f079905a6a76483682af4fb9dadc4e58', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['86bc3a84-a32a-4900-93e6-09d62ba875c3'], 'standard_attr_id': 892, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:29:42Z', 'updated_at': '2026-06-08T14:29:42Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 13129}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.065235 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-9a4f3019-f88a-411b-a449-099a5c655333, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestShelveInstance-458690823-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.065572 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-1b5ccc58-ebbd-4a67-a485-e6aceb90f4d2, external_ids_update=None, columns={'tag': 13129}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.065679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.068856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:59.068856 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.068856 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:59.068856 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:59.068856 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.068856 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:59.068856 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.068856 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:59.068856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 9a4f3019-f88a-411b-a449-099a5c655333 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.068856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:59.068856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:30:59.068856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:30:59.068856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:30:59.068856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:30:59.069127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:30:59.069127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:59.069127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.069127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.069127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:59.069127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:59.069127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.069127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.069127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:59.069127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:59.069127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:59.069127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:59.069127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:59.069346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:59.069346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:59.069346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.069346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:59.069346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:59.069346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.069346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:59.069346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.069346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:59.069346 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource cf2f1f8a-00e6-41b3-8c1a-d1da3e956b82 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:59.111329 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-cf2f1f8a-00e6-41b3-8c1a-d1da3e956b82, resource={'id': 'cf2f1f8a-00e6-41b3-8c1a-d1da3e956b82', 'name': 'tempest-AttachVolumeShelveTestJSON-192128213-network', 'tenant_id': '39c03fc0701643f6a33b3ce47bfc68c9', 'project_id': '39c03fc0701643f6a33b3ce47bfc68c9', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['96984df7-79ae-4d78-8c18-bafcea970f2f'], 'standard_attr_id': 603, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:27:00Z', 'updated_at': '2026-06-08T14:27:00Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 935}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.111523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-cf2f1f8a-00e6-41b3-8c1a-d1da3e956b82, col_values=(('external_ids', {'neutron:network_name': 'tempest-AttachVolumeShelveTestJSON-192128213-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.111708 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-76336934-f3b3-4305-bf03-ec0e10f79932, external_ids_update=None, columns={'tag': 935}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.111828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.112270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:59.112270 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.112270 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:59.112270 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:59.112270 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.112270 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:59.112270 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.112270 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:59.113060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource cf2f1f8a-00e6-41b3-8c1a-d1da3e956b82 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.113060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:59.113060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:30:59.113060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:30:59.113060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:30:59.113060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:30:59.113060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:30:59.113060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:59.113060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.113060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.113060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:59.113442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:59.113442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.113442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.113442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:59.113442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:59.113442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:59.113442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:59.113442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:59.113442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:59.113442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:59.113442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.113442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:59.113442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:59.113442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.113833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:59.113833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.113833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:59.113833 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 2af03962-3944-44a0-8e74-7d1416c723a3 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:59.140384 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-2af03962-3944-44a0-8e74-7d1416c723a3, resource={'id': '2af03962-3944-44a0-8e74-7d1416c723a3', 'name': 'tempest-ServerRescueNegativeTestJSON-1577350096-network', 'tenant_id': 'dcc7c2372ef74d6d93f8dd7e26d5b5ca', 'project_id': 'dcc7c2372ef74d6d93f8dd7e26d5b5ca', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['b3780f8f-a594-491c-9b96-b1d5dcf18ca9'], 'standard_attr_id': 810, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:28:38Z', 'updated_at': '2026-06-08T14:28:38Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 18619}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.148243 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-2af03962-3944-44a0-8e74-7d1416c723a3, col_values=(('external_ids', {'neutron:network_name': 'tempest-ServerRescueNegativeTestJSON-1577350096-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.148536 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-8369e7e1-d569-4733-9885-fc23cabb8a44, external_ids_update=None, columns={'tag': 18619}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.148622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.149114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:59.149114 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.149114 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:59.149114 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:59.149114 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.149114 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:59.149114 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.149114 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:59.149855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 2af03962-3944-44a0-8e74-7d1416c723a3 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.149855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:59.149855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:30:59.149855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:30:59.149855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:30:59.149855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:30:59.149855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:30:59.149855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:59.149855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.149855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.149855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:59.150149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:59.150149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.150149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.150149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:59.150149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:59.150149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:59.150149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:59.150149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:59.150149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:59.150149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:59.150149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.150149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:59.150149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:59.150149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.150530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:59.150530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.150530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:59.150530 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 89a49737-853e-427c-af23-b03c6a22ee04 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:59.189581 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-89a49737-853e-427c-af23-b03c6a22ee04, resource={'id': '89a49737-853e-427c-af23-b03c6a22ee04', 'name': 'tempest-VolumesBackupsTest-1869380095-network', 'tenant_id': '170c5fa9717d4fedab1978b73c803d23', 'project_id': '170c5fa9717d4fedab1978b73c803d23', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['5fd461b5-c0e3-4027-9737-b1373e1e2410'], 'standard_attr_id': 865, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:29:26Z', 'updated_at': '2026-06-08T14:29:27Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 20867}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.189787 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-89a49737-853e-427c-af23-b03c6a22ee04, col_values=(('external_ids', {'neutron:network_name': 'tempest-VolumesBackupsTest-1869380095-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.190047 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-957ea1cb-cce6-4934-9e2f-67eb41545cb9, external_ids_update=None, columns={'tag': 20867}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.190136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.190574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:59.190574 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.190574 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:59.190574 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:59.190574 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.190574 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:59.190574 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.190574 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:59.191314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 89a49737-853e-427c-af23-b03c6a22ee04 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.191314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:59.191314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:30:59.191314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:30:59.191314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:30:59.191314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:30:59.191314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:30:59.191314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:59.191314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.191314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.191314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:59.191503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:59.191503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.191503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.191503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:59.191503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:59.191503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:59.191503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:59.191503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:59.191503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:59.191503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:59.191503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.191503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:59.191503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:59.191503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.191727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:59.191727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.191727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:59.191727 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 227858ec-9624-4ffa-a9a6-653d1d276ea5 (type: security_groups) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:59.226362 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Successfully bumped revision number for resource 227858ec-9624-4ffa-a9a6-653d1d276ea5 (type: security_groups) to 7 Jun 08 14:30:59.226494 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource b47ec7fc-2f29-469e-b493-52a3d0510bc4 (type: security_groups) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:59.254165 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Successfully bumped revision number for resource b47ec7fc-2f29-469e-b493-52a3d0510bc4 (type: security_groups) to 3 Jun 08 14:30:59.254317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 851b2d98-534b-406f-916e-d110ce729060 (type: security_groups) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:59.278770 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Successfully bumped revision number for resource 851b2d98-534b-406f-916e-d110ce729060 (type: security_groups) to 3 Jun 08 14:30:59.278925 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 5d47963e-e0bb-4b19-b909-213830d8de54 (type: security_groups) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:59.304366 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Successfully bumped revision number for resource 5d47963e-e0bb-4b19-b909-213830d8de54 (type: security_groups) to 3 Jun 08 14:30:59.304552 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource f6c5b4d8-5e07-474a-aeff-ac031de0ac47 (type: security_groups) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:59.337371 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Successfully bumped revision number for resource f6c5b4d8-5e07-474a-aeff-ac031de0ac47 (type: security_groups) to 3 Jun 08 14:30:59.337561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource e18214a6-42af-4a82-ae91-b005b582cf79 (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:59.435449 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, resource={'id': 'e18214a6-42af-4a82-ae91-b005b582cf79', 'name': 'tempest-ServerRescueNegativeTestJSON-1577350096-router', 'project_id': 'dcc7c2372ef74d6d93f8dd7e26d5b5ca', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.167'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::cc'}], 'enable_snat': True}, 'gw_port_id': 'e15c0ec6-020b-4475-81bf-582cec4587be', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.167', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::cc', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:28:39Z', 'updated_at': '2026-06-08T14:28:41Z', 'revision_number': 3, 'tenant_id': 'dcc7c2372ef74d6d93f8dd7e26d5b5ca'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.435718 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, columns={'external_ids': {'neutron:router_name': 'tempest-ServerRescueNegativeTestJSON-1577350096-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.435965 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "e15c0ec6-020b-4475-81bf-582cec4587be"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.436147 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "e15c0ec6-020b-4475-81bf-582cec4587be"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.436304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.436918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:59.436918 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.436918 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:59.436918 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:59.436918 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.436918 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:59.436918 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.436918 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:59.437129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router e18214a6-42af-4a82-ae91-b005b582cf79. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.438201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource e18214a6-42af-4a82-ae91-b005b582cf79 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.438201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:59.438201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:30:59.438201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:30:59.438201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:30:59.438201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:30:59.438201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:30:59.438201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:30:59.438201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:30:59.438201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:30:59.438201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:30:59.438625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:30:59.438625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:30:59.438625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:59.438625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.438625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.438625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:59.438625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:59.438625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.438625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.438625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:59.438625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:59.438625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:59.438625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:59.438625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:59.438914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:59.438914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:59.438914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.438914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:59.438914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:59.438914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.438914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:59.438914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.438914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:59.438914 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:59.519756 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, resource={'id': 'e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c', 'name': 'tempest-VolumesBackupsTest-1869380095-router', 'project_id': '170c5fa9717d4fedab1978b73c803d23', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.197'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::1d5'}], 'enable_snat': True}, 'gw_port_id': 'f8c22fde-88ff-45d3-b601-a8438ea13038', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.197', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::1d5', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:29:28Z', 'updated_at': '2026-06-08T14:29:31Z', 'revision_number': 3, 'tenant_id': '170c5fa9717d4fedab1978b73c803d23'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.520032 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, columns={'external_ids': {'neutron:router_name': 'tempest-VolumesBackupsTest-1869380095-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.520241 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "f8c22fde-88ff-45d3-b601-a8438ea13038"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.520392 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "f8c22fde-88ff-45d3-b601-a8438ea13038"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.520581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.521686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:59.521686 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.521686 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:59.521686 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:59.521686 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.521686 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:59.521686 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.521686 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:59.521686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.522698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.522698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:59.522698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:30:59.522698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:30:59.522698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:30:59.522698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:30:59.522698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:30:59.522698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:30:59.522698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:30:59.522698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:30:59.522698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:30:59.522934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:30:59.522934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:30:59.522934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:59.522934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.522934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.522934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:59.522934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:59.522934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.522934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.522934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:59.522934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:59.522934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:59.522934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:59.522934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:59.523180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:59.523180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:59.523180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.523180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:59.523180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:59.523180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.523180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:59.523180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.523180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:59.523180 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 30bf0b43-acea-45e6-9fcd-f04d92d84af8 (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:59.609476 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, resource={'id': '30bf0b43-acea-45e6-9fcd-f04d92d84af8', 'name': 'tempest-AttachVolumeShelveTestJSON-192128213-router', 'project_id': '39c03fc0701643f6a33b3ce47bfc68c9', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.32'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::3e8'}], 'enable_snat': True}, 'gw_port_id': 'b09910e6-6878-40cb-a995-7f3c9a043709', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.32', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::3e8', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:27:01Z', 'updated_at': '2026-06-08T14:27:03Z', 'revision_number': 3, 'tenant_id': '39c03fc0701643f6a33b3ce47bfc68c9'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.609696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, columns={'external_ids': {'neutron:router_name': 'tempest-AttachVolumeShelveTestJSON-192128213-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.609879 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "b09910e6-6878-40cb-a995-7f3c9a043709"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.610015 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "b09910e6-6878-40cb-a995-7f3c9a043709"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.610112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.610603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:59.610603 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.610603 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:59.610603 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:59.610603 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.610603 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:59.610603 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.610603 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:59.610881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router 30bf0b43-acea-45e6-9fcd-f04d92d84af8. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.613026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 30bf0b43-acea-45e6-9fcd-f04d92d84af8 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.613026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:59.613026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:30:59.613026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:30:59.613026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:30:59.613026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:30:59.613026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:30:59.613026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:30:59.613026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:30:59.613026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:30:59.613026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:30:59.613410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:30:59.613410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:30:59.613410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:59.613410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.613410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.613410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:59.613410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:59.613410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.613410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.613410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:59.613410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:59.613410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:59.613410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:59.613410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:59.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:59.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:59.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:59.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:59.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:59.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:59.613726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource fb66c1ba-ca61-4eca-a52b-e1bd5aacc770 (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:59.695316 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, resource={'id': 'fb66c1ba-ca61-4eca-a52b-e1bd5aacc770', 'name': 'tempest-TestShelveInstance-458690823-router', 'project_id': 'f079905a6a76483682af4fb9dadc4e58', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.54'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::266'}], 'enable_snat': True}, 'gw_port_id': '23d8feb6-6004-41e4-a5fe-404b0919d121', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.54', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::266', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:29:43Z', 'updated_at': '2026-06-08T14:29:45Z', 'revision_number': 3, 'tenant_id': 'f079905a6a76483682af4fb9dadc4e58'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.695806 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, columns={'external_ids': {'neutron:router_name': 'tempest-TestShelveInstance-458690823-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.695806 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "23d8feb6-6004-41e4-a5fe-404b0919d121"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.695806 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "23d8feb6-6004-41e4-a5fe-404b0919d121"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.695806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.695806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:59.695806 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.695806 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:59.695806 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:59.696039 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.696039 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:59.696039 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.696039 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:59.696039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router fb66c1ba-ca61-4eca-a52b-e1bd5aacc770. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.696039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource fb66c1ba-ca61-4eca-a52b-e1bd5aacc770 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.696039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:59.696039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:30:59.696039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:30:59.696039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:30:59.696039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:30:59.696320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:30:59.696320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:30:59.696320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:30:59.696320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:30:59.696320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:30:59.696320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:30:59.696320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:30:59.696320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:59.696320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.696320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.696320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:59.696320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:59.696320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.696582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.696582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:59.696582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:59.696582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:59.696582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:59.696582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:59.696582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:59.696582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:59.696582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.696582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:59.696582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:59.696582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.696582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:59.696818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.696818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:59.696818 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource acd0c4a8-21a2-4e59-8c73-9c6fb220d996 (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:59.790706 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, resource={'id': 'acd0c4a8-21a2-4e59-8c73-9c6fb220d996', 'name': 'tempest-TestShelveInstance-1091677164-router', 'project_id': '95d135317c314a42bb44fd26ddacaefd', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.116'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::14a'}], 'enable_snat': True}, 'gw_port_id': 'f81cc546-e1a3-499a-b700-405a8e7bb66b', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.116', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::14a', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:29:48Z', 'updated_at': '2026-06-08T14:29:51Z', 'revision_number': 3, 'tenant_id': '95d135317c314a42bb44fd26ddacaefd'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.790944 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, columns={'external_ids': {'neutron:router_name': 'tempest-TestShelveInstance-1091677164-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.791171 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "f81cc546-e1a3-499a-b700-405a8e7bb66b"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.791341 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "f81cc546-e1a3-499a-b700-405a8e7bb66b"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.791452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.792059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:59.792059 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.792059 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:59.792059 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:59.792059 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.792059 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:59.792059 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.792059 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:59.792348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router acd0c4a8-21a2-4e59-8c73-9c6fb220d996. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.793826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource acd0c4a8-21a2-4e59-8c73-9c6fb220d996 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.793826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:59.793826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:30:59.793826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:30:59.793826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:30:59.793826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:30:59.793826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:30:59.793826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:30:59.793826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:30:59.793826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:30:59.793826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:30:59.794151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:30:59.794151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:30:59.794151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:59.794151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.794151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.794151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:59.794151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:59.794151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.794151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.794151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:59.794151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:59.794151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:59.794151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:59.794151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:59.794465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:59.794465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:59.794465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.794465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:59.794465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:59.794465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.794465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:59.794465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.794465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:59.794465 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource dbcb05ba-3761-4282-b616-71a4937f4fed (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:59.872837 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, resource={'id': 'dbcb05ba-3761-4282-b616-71a4937f4fed', 'name': 'tempest-ServerActionsTestOtherA-1380864152-router', 'project_id': '2ce3c80c0935406eba76da06872272d9', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.8'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::3b2'}], 'enable_snat': True}, 'gw_port_id': 'dab47abd-ef9d-4701-9831-0872332585b4', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.8', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::3b2', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:26:48Z', 'updated_at': '2026-06-08T14:26:50Z', 'revision_number': 3, 'tenant_id': '2ce3c80c0935406eba76da06872272d9'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.873069 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, columns={'external_ids': {'neutron:router_name': 'tempest-ServerActionsTestOtherA-1380864152-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.873243 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "dab47abd-ef9d-4701-9831-0872332585b4"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.873387 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "dab47abd-ef9d-4701-9831-0872332585b4"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.873500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.874103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:59.874103 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.874103 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:59.874103 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:59.874103 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.874103 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:59.874103 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.874103 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:59.874521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router dbcb05ba-3761-4282-b616-71a4937f4fed. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.876150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource dbcb05ba-3761-4282-b616-71a4937f4fed (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.876150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:59.876150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:30:59.876150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:30:59.876150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:30:59.876150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:30:59.876150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:30:59.876150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:30:59.876150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:30:59.876150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:30:59.876150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:30:59.876385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:30:59.876385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:30:59.876385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:59.876385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.876385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.876385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:59.876385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:59.876385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.876385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.876385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:59.876385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:59.876385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:59.876385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:59.876385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:59.876629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:59.876629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:59.876629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.876629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:59.876629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:59.876629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.876629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:59.876629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.876629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:59.876629 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource e15c0ec6-020b-4475-81bf-582cec4587be (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:30:59.972298 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-e15c0ec6-020b-4475-81bf-582cec4587be, resource={'id': 'e15c0ec6-020b-4475-81bf-582cec4587be', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:53:9b:38', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'e18214a6-42af-4a82-ae91-b005b582cf79', 'device_owner': 'network:router_gateway', 'standard_attr_id': 815, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.167'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::cc'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:28:39Z', 'updated_at': '2026-06-08T14:28:41Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.972540 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-e15c0ec6-020b-4475-81bf-582cec4587be, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-e18214a6-42af-4a82-ae91-b005b582cf79'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.167/24', '2001:db8::cc/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.972794 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=e15c0ec6-020b-4475-81bf-582cec4587be, lrouter_port=lrp-e15c0ec6-020b-4475-81bf-582cec4587be, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:30:59.972960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.973541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:30:59.973541 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.973541 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:30:59.973541 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:30:59.973541 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.973541 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:30:59.973541 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.973541 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:30:59.974724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource e15c0ec6-020b-4475-81bf-582cec4587be (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.974724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:30:59.974724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:30:59.974724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:30:59.974724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:30:59.974724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:30:59.974724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:30:59.974724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:30:59.974724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.974724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.974724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:30:59.975130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:30:59.975130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:30:59.975130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:30:59.975130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:30:59.975130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:30:59.975130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:30:59.975130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:30:59.975130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:30:59.975130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:30:59.975130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:30:59.975130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:30:59.975130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:30:59.975130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:30:59.975130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:30:59.975502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:30:59.975502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:30:59.975502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:30:59.975502 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource f8c22fde-88ff-45d3-b601-a8438ea13038 (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:31:00.098071 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-f8c22fde-88ff-45d3-b601-a8438ea13038, resource={'id': 'f8c22fde-88ff-45d3-b601-a8438ea13038', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:1b:7a:09', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c', 'device_owner': 'network:router_gateway', 'standard_attr_id': 870, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.197'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::1d5'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:29:28Z', 'updated_at': '2026-06-08T14:29:31Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.098388 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-f8c22fde-88ff-45d3-b601-a8438ea13038, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.197/24', '2001:db8::1d5/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.098733 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=f8c22fde-88ff-45d3-b601-a8438ea13038, lrouter_port=lrp-f8c22fde-88ff-45d3-b601-a8438ea13038, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.098989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.099700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:00.099700 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.099700 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:00.099700 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:00.099700 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.099700 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:00.099700 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.099700 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:00.100620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource f8c22fde-88ff-45d3-b601-a8438ea13038 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.100620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:00.100620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:31:00.100620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:31:00.100620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:31:00.100620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:31:00.100620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:31:00.100620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:00.100620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.100620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:00.100620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:00.100835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:00.100835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.100835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:00.100835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:00.100835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:00.100835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:00.100835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:00.100835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:00.100835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:00.100835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:00.100835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.100835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:00.100835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:00.100835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.101136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:00.101136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.101136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:00.101136 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource b09910e6-6878-40cb-a995-7f3c9a043709 (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:31:00.184236 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-b09910e6-6878-40cb-a995-7f3c9a043709, resource={'id': 'b09910e6-6878-40cb-a995-7f3c9a043709', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:30:aa:ac', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '30bf0b43-acea-45e6-9fcd-f04d92d84af8', 'device_owner': 'network:router_gateway', 'standard_attr_id': 609, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.32'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::3e8'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:27:01Z', 'updated_at': '2026-06-08T14:27:03Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.184569 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-b09910e6-6878-40cb-a995-7f3c9a043709, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.32/24', '2001:db8::3e8/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.184890 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=b09910e6-6878-40cb-a995-7f3c9a043709, lrouter_port=lrp-b09910e6-6878-40cb-a995-7f3c9a043709, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.185129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.185722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:00.185722 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.185722 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:00.185722 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:00.185722 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.185722 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:00.185722 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.185722 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:00.186561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource b09910e6-6878-40cb-a995-7f3c9a043709 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.186561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:00.186561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:31:00.186561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:31:00.186561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:31:00.186561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:31:00.186561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:31:00.186561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:00.186561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.186561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:00.186561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:00.186799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:00.186799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.186799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:00.186799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:00.186799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:00.186799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:00.186799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:00.186799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:00.186799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:00.186799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:00.186799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.186799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:00.186799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:00.186799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.187055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:00.187055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.187055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:00.187055 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 23d8feb6-6004-41e4-a5fe-404b0919d121 (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:31:00.276996 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-23d8feb6-6004-41e4-a5fe-404b0919d121, resource={'id': '23d8feb6-6004-41e4-a5fe-404b0919d121', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:22:65:b1', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'fb66c1ba-ca61-4eca-a52b-e1bd5aacc770', 'device_owner': 'network:router_gateway', 'standard_attr_id': 898, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.54'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::266'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:29:43Z', 'updated_at': '2026-06-08T14:29:45Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.277283 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-23d8feb6-6004-41e4-a5fe-404b0919d121, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::266/64', '172.24.5.54/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.277556 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=23d8feb6-6004-41e4-a5fe-404b0919d121, lrouter_port=lrp-23d8feb6-6004-41e4-a5fe-404b0919d121, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.277738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.278313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:00.278313 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.278313 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:00.278313 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:00.278313 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.278313 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:00.278313 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.278313 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:00.279196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 23d8feb6-6004-41e4-a5fe-404b0919d121 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.279196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:00.279196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:31:00.279196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:31:00.279196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:31:00.279196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:31:00.279196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:31:00.279196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:00.279196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.279196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:00.279196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:00.279498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:00.279498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.279498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:00.279498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:00.279498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:00.279498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:00.279498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:00.279498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:00.279498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:00.279498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:00.279498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.279498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:00.279498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:00.279498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.279781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:00.279781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.279781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:00.279781 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource dab47abd-ef9d-4701-9831-0872332585b4 (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:31:00.378187 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-dab47abd-ef9d-4701-9831-0872332585b4, resource={'id': 'dab47abd-ef9d-4701-9831-0872332585b4', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:f1:90:f8', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'dbcb05ba-3761-4282-b616-71a4937f4fed', 'device_owner': 'network:router_gateway', 'standard_attr_id': 579, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.8'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::3b2'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:26:48Z', 'updated_at': '2026-06-08T14:26:51Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.378422 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-dab47abd-ef9d-4701-9831-0872332585b4, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-dbcb05ba-3761-4282-b616-71a4937f4fed'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::3b2/64', '172.24.5.8/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.378635 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=dab47abd-ef9d-4701-9831-0872332585b4, lrouter_port=lrp-dab47abd-ef9d-4701-9831-0872332585b4, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.378797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.379319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:00.379319 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.379319 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:00.379319 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:00.379319 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.379319 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:00.379319 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.379319 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:00.380163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource dab47abd-ef9d-4701-9831-0872332585b4 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.380163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:00.380163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:31:00.380163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:31:00.380163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:31:00.380163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:31:00.380163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:31:00.380163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:00.380163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.380163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:00.380163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:00.380396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:00.380396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.380396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:00.380396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:00.380396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:00.380396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:00.380396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:00.380396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:00.380396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:00.380396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:00.380396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.380396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:00.380396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:00.380396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.380621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:00.380621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.380621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:00.380621 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource f81cc546-e1a3-499a-b700-405a8e7bb66b (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:31:00.509149 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-f81cc546-e1a3-499a-b700-405a8e7bb66b, resource={'id': 'f81cc546-e1a3-499a-b700-405a8e7bb66b', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:02:88:7a', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'acd0c4a8-21a2-4e59-8c73-9c6fb220d996', 'device_owner': 'network:router_gateway', 'standard_attr_id': 910, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.116'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::14a'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:29:48Z', 'updated_at': '2026-06-08T14:29:51Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.509419 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-f81cc546-e1a3-499a-b700-405a8e7bb66b, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.116/24', '2001:db8::14a/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.509678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=f81cc546-e1a3-499a-b700-405a8e7bb66b, lrouter_port=lrp-f81cc546-e1a3-499a-b700-405a8e7bb66b, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.509852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.510406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:00.510406 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.510406 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:00.510406 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:00.510406 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.510406 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:00.510406 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.510406 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:00.511200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource f81cc546-e1a3-499a-b700-405a8e7bb66b (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.511200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:00.511200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:31:00.511200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:31:00.511200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:31:00.511200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:31:00.511200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:31:00.511200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:00.511200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.511200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:00.511200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:00.511529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:00.511529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.511529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:00.511529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:00.511529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:00.511529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:00.511529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:00.511529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:00.511529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:00.511529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:00.511529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.511529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:00.511529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:00.511529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.511941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:00.511941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.511941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:00.511941 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 49bacaa1-8ede-4596-ba0a-64ecc3a9a217 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.511941 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(49bacaa1-8ede-4596-ba0a-64ecc3a9a217) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.516265 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource e3a20595-6524-4ee0-98bd-84dbf5a6f701 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.516533 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(e3a20595-6524-4ee0-98bd-84dbf5a6f701) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.520036 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource b87021c5-89b6-4d48-8446-58e414835d2b (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.520258 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(b87021c5-89b6-4d48-8446-58e414835d2b) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.524386 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource b4f82672-8864-41ec-8bef-d90ba72b7280 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.524646 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(b4f82672-8864-41ec-8bef-d90ba72b7280) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.529783 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource aa60d49a-485a-4683-81bf-0afb43ede288 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.530059 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(aa60d49a-485a-4683-81bf-0afb43ede288) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.534403 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 98f87ce8-1f5e-4abd-b6de-32f21dbb1d4b (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.534403 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(98f87ce8-1f5e-4abd-b6de-32f21dbb1d4b) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.538196 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 8d807824-aa5b-4eaf-86be-c81ac23b66f7 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.538456 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(8d807824-aa5b-4eaf-86be-c81ac23b66f7) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.542578 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 8abc923a-54cc-4a1b-884b-e788ad7fdb68 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.542800 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(8abc923a-54cc-4a1b-884b-e788ad7fdb68) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.546287 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 861b2ae6-e612-419a-a9f7-1846b04360bb (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.546486 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(861b2ae6-e612-419a-a9f7-1846b04360bb) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.551095 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 7e30389c-1ed3-4581-91b8-fe46c4ae387f (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.551341 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(7e30389c-1ed3-4581-91b8-fe46c4ae387f) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.555032 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 01a9b8ad-3ff6-4b09-a106-d61f3935a149 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.555314 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(01a9b8ad-3ff6-4b09-a106-d61f3935a149) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.559360 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 3e24f5ab-3d25-426f-ad2c-d2734e8e5c7c (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.559674 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(3e24f5ab-3d25-426f-ad2c-d2734e8e5c7c) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.564808 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 37b1707e-d3f2-4888-8a3f-ee37f8f220a8 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.565155 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(37b1707e-d3f2-4888-8a3f-ee37f8f220a8) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.569694 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 34a4b1c2-92b6-418a-9531-a72306886469 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.569939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(34a4b1c2-92b6-418a-9531-a72306886469) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.574739 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 2c07b3a1-0113-4819-af6f-6802d7d76d11 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.575063 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(2c07b3a1-0113-4819-af6f-6802d7d76d11) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.579447 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 1d40de7d-5b17-483f-8b89-e7bef0bdec58 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.579639 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(1d40de7d-5b17-483f-8b89-e7bef0bdec58) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.583786 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 1d3e480b-c026-42ee-a0fa-648117314a93 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.584083 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(1d3e480b-c026-42ee-a0fa-648117314a93) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.589365 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 14c69157-2348-47e8-b08f-512c2590d948 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.589365 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(14c69157-2348-47e8-b08f-512c2590d948) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.596803 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 10fbd62b-95da-42f8-b0dd-b6aed7b84e48 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.597132 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(10fbd62b-95da-42f8-b0dd-b6aed7b84e48) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.602359 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 0e931a04-5c97-41db-bec5-04ca69f988a0 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:31:00.602658 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(0e931a04-5c97-41db-bec5-04ca69f988a0) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:31:00.610171 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_for_inconsistencies finished in 1.750 seconds Jun 08 14:31:00.754745 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:00.756745 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:00.756745 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:00.757035 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.757194 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.757373 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.757481 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.757605 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.757714 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.758092 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.758238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.758390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.758534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.761156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:00.761156 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.761156 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:00.761156 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:00.761156 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.761156 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:00.761156 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.761156 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:00.761156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.761156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:00.761156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:00.761156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:00.761156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.761156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:00.761156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:00.761731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.761731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:00.761731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:00.761731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:00.761731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.761731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:00.761731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:00.761731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:00.761731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:00.761731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:00.761731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:00.761731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:00.761731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:00.761731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.762118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:00.762118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:00.762118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.762118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:00.762118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.762118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:00.762118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.762118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:00.762118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:00.762118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:00.762118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:00.762118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:00.762118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.762118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:00.762506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:00.763001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:00.763001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:00.763001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:00.763001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.763001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:00.763001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:00.763001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.763001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:00.763001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.763001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:00.777829 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:00.780802 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:00.781032 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:00.781228 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:31:00.781698 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.781901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.782763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:00.782763 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.782763 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:00.782763 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:00.782763 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.782763 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:00.782763 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.782763 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:00.783709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.783709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:00.783709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:00.783709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:00.783709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.783709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:00.783709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:00.783709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:00.783709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.783709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:00.783709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:00.783709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:00.783709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:00.783709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:00.783709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:00.784162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:00.784162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:00.784162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.784162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:00.784162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:00.784162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.784162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:00.784162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.784162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:00.784631 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:00.786334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.786334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:00.786334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:00.786334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:00.786334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.786334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:00.786334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:00.786334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:31:00.786334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:00.786334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:00.786334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.786334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:00.786804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:00.786804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:00.786804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.786804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:00.786804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:00.786804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:00.786804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:00.786804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:00.786804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:00.786804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:00.786804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:00.786804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.786804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:00.786804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:00.787058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.787058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:00.787058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.787058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:00.787058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.787058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:00.787058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:00.787058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:00.787058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:00.787058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:00.787058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.787058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:00.787058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:00.787058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.787397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:00.787397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:00.787397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.787397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:00.787397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:00.787397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.787397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:00.787397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:00.787397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:31:00.787397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:00.787397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:00.787397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.787397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:00.787397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:00.787397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:00.787397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.787397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:00.787397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:00.787397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:00.787699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:00.787699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:00.787699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:00.787699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:00.787699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:00.787699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.787699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:00.787699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:00.787699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.787699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:00.787699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.787699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:00.787699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:00.787699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:00.787916 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:00.787916 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.788164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.788548 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:00.788737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:00.788737 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.788737 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:00.788737 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:00.788737 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.788737 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:00.788737 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.788737 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:00.789308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.789308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:00.789308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:00.789308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:00.789308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.789308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:00.789308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:00.789308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:00.789308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.789308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:00.789308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:00.789308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:00.789308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:00.789308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:00.789308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:00.789583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:00.789583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:00.789583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.789583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:00.789583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:00.789583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.789583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:00.789583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.789583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:00.790049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.790049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:00.790049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:00.790049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:00.790049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.790049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:00.790049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:00.790049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:00.790049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:00.790049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.790049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:00.790337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:00.790337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:00.790337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.790337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:00.790337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:00.790337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:00.790337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:00.790337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:00.790337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:00.790337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:00.790337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:00.790337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.790337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:00.790337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:00.790568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.790568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:00.790568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.790568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:00.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:00.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:00.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:00.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:00.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:00.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:00.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:00.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:00.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:00.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:00.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:00.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.790919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:00.791267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:00.791267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:00.791267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:00.791267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.791267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:00.791267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:00.791267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:00.791267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.791267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:00.791267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:00.791267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:00.791267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:00.791267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:00.791267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:00.791267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:00.791267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:00.791267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.791267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:00.791267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:00.791590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.791590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:00.791590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.791590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:00.791590 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:00.791590 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:00.791590 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:00.791822 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.792019 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:00.792342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.792812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:00.792812 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.792812 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:00.792812 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:00.792812 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.792812 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:00.792812 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.792812 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:00.793414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.793414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:00.793414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:00.793414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:00.793414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.793414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:00.793414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:00.793414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.793414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:00.793414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:00.793414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:00.793414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.793689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:00.793689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:00.793689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:00.793689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:00.793689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:00.793689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:00.793689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:00.793689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:00.793689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.793689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:00.793689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:00.793689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.793689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:00.793929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.793929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:00.794136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.794136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:00.794136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:00.794136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:00.794136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:00.794136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:00.794136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.794136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:00.794136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:00.794136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.794136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:00.794136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:00.794136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.794136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:00.794136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:00.794136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:00.794490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:00.794490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:00.794490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.794490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:00.794490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:00.794490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:00.794490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:00.794490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:00.794490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:00.794490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:00.794490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:00.794490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:00.794490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:00.794490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:00.794490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:00.794490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:00.794490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:00.794490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:00.794490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:00.794772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:00.794772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:00.794772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:03.679390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:03.679807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:03.679869 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:03.680555 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:03.719685 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:03.719932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:03.720557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:03.720557 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:03.720557 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:03.720557 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:03.720557 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:03.720557 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:03.720557 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:03.720557 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:03.721785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:03.721785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:03.721785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:03.721785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:03.721785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:03.721785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:03.721785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:03.721785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:03.721785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:03.721785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:03.721785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:03.721785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:03.722183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:03.722183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:03.722183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:03.722183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:03.722183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:03.722183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:03.722183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:03.722183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:03.722183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:03.722183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:03.722183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:03.722183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:03.722183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:03.722584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:03.722584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:03.723058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:03.723058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:03.723058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:03.723058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:03.723058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:03.723058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:03.723058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:03.723058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:03.723058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:03.723058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:03.723058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:03.723058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:03.723058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:03.723058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:03.723058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:03.723058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:03.723704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:03.723704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:03.723704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:03.723704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:03.723704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:03.723704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:03.723704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:03.723704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:03.723704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:03.723704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:03.723704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:03.723704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:03.723704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:03.723704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:03.723704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:03.723704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:03.723704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:03.723704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:03.723704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:03.724182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:03.724182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:03.724182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:05.755807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:05.756400 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:05.756400 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:05.758646 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:05.758646 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:05.758646 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:05.758646 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:05.758646 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:05.759073 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:05.759073 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:05.759073 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:05.759073 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:05.759073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.759073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:05.759073 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:05.759073 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:05.759073 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:05.759405 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:05.759405 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:05.759405 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.759405 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:05.761435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.761435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:05.761435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:05.761435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:05.761435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.761435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:05.761435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:05.761435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.761435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:05.761435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:05.761435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:05.761435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.761821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:05.761821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:05.761821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:05.761821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:05.761821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:05.761821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:05.761821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:05.761821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:05.761821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:05.761821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:05.761821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:05.761821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:05.761821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:05.762238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.762238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:05.762600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.762600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:05.762600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:05.762600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:05.762600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:05.762600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:05.762600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.762600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:05.762600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:05.762600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.762600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:05.762600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:05.762600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.762600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:05.762600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:05.762600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.763043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:05.763043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:05.763043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.763043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:05.763043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:05.763043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:05.763043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.763043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:05.763043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:05.763043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:05.763043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:05.763043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:05.763043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:05.763043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:05.763043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:05.763043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:05.763043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:05.763043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:05.763043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:05.763548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:05.763548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.763548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:05.780808 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:05.781059 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:05.781141 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:05.781229 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:31:05.782924 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:05.782924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.783114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:05.783114 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:05.783114 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:05.783114 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:05.783114 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:05.783114 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:05.783114 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.783114 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:05.783599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.783599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:05.783599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:05.783599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:05.783599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.783599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:05.783599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:05.783599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:05.783599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.783599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:05.783599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:05.783599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:05.783599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:05.783599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:05.783599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:05.783916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:05.783916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:05.783916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:05.783916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:05.783916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:05.783916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:05.783916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:05.783916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.783916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:05.784243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.784243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:05.784243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:05.784243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:05.784243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.784243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:05.784243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:05.784243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:31:05.784243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:05.784243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:05.784243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.784243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:05.785473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:05.785473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:05.785473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.785473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:05.785473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:05.785473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:05.785473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:05.785473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:05.785473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:05.785473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:05.785473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:05.785473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:05.785473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:05.785473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:05.785843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:05.785843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:05.785843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.785843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:05.785843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.785843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:05.785843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:05.785843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:05.785843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:05.785843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:05.785843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.785843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:05.785843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:05.785843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.786131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:05.786131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:05.786131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.786131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:05.786131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:05.786131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.786131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:05.786131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:05.786131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:31:05.786131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:05.786131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:05.786131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.786131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:05.786131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:05.786131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:05.786131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.786131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:05.786131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:05.786131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:05.786473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:05.786473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:05.786473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:05.786473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:05.786473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:05.786473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:05.786473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:05.786473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:05.786473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:05.786473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:05.786473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.786473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:05.787034 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:05.787817 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:05.787900 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:05.788024 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:05.788377 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:05.788634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:05.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:05.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:05.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:05.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:05.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:05.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.789134 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:05.790815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.790815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:05.790815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:05.790815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:05.790815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.790815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:05.790815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:05.790815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:05.790815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.790815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:05.790815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:05.790815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:05.790815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:05.790815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:05.790815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:05.791115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:05.791115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:05.791115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:05.791115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:05.791115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:05.791115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:05.791115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:05.791115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.791115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:05.791703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.791703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:05.791703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:05.791703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:05.791703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.791703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:05.791703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:05.791703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:05.791703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:05.791703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.791703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:05.791936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:05.791936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:05.791936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.791936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:05.791936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:05.791936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:05.791936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:05.791936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:05.791936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:05.791936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:05.791936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:05.791936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:05.791936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:05.791936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:05.792168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:05.792168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:05.792168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.792168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:05.792672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.792672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:05.792672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:05.792672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:05.792672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:05.792672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:05.792672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.792672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:05.792672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:05.792672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.792672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:05.792672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:05.792672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.792672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:05.792672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:05.792672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.792672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:05.793011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:05.793011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:05.793011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:05.793011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.793011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:05.793011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:05.793011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:05.793011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.793011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:05.793011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:05.793011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:05.793011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:05.793011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:05.793011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:05.793011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:05.793011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:05.793011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:05.793011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:05.793011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:05.793366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:05.793366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:05.793366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.793366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:05.793366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:05.793366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:05.793366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:05.793506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:05.793881 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:05.794130 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:05.794376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.794849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:05.794849 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:05.794849 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:05.794849 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:05.794849 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:05.794849 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:05.794849 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.794849 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:05.795502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.795502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:05.795502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:05.795502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:05.795502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.795502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:05.795502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:05.795502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.795502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:05.795502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:05.795502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:05.795502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.795716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:05.795716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:05.795716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:05.795716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:05.795716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:05.795716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:05.795716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:05.795716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:05.795716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:05.795716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:05.795716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:05.795716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:05.795716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:05.795938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.795938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:05.796139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.796139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:05.796139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:05.796139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:05.796139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:05.796139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:05.796139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.796139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:05.796139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:05.796139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.796139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:05.796139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:05.796139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.796139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:05.796139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:05.796139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:05.796425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:05.796425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:05.796425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.796425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:05.796425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:05.796425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:05.796425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:05.796425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:05.796425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:05.796425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:05.796425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:05.796425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:05.796425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:05.796425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:05.796425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:05.796425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:05.796425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:05.796425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:05.796425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:05.796790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:05.796790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:05.796790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:08.679792 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:08.680960 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:08.680960 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:08.680960 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:08.726700 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:08.726916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:08.727762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:08.727762 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:08.727762 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:08.727762 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:08.727762 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:08.727762 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:08.727762 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:08.727762 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:08.728387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:08.728387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:08.728387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:08.728387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:08.728387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:08.728387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:08.728387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:08.728387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:08.728387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:08.728387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:08.728387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:08.728387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:08.728630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:08.728630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:08.728630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:08.728630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:08.728630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:08.728630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:08.728630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:08.728630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:08.728630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:08.728630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:08.728630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:08.728630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:08.728630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:08.728844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:08.728844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:08.729088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:08.729088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:08.729088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:08.729088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:08.729088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:08.729088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:08.729088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:08.729088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:08.729088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:08.729088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:08.729088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:08.729088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:08.729088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:08.729088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:08.729088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:08.729088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:08.729393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:08.729393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:08.729393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:08.729393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:08.729393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:08.729393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:08.729393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:08.729393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:08.729393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:08.729393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:08.729393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:08.729393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:08.729393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:08.729393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:08.729393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:08.729393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:08.729393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:08.729393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:08.729393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:08.729703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:08.729703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:08.729703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:10.756211 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:10.759523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:10.759592 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:10.760558 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:10.760754 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:10.760884 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:10.761031 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:10.761148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:10.761293 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:10.761418 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:10.761533 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:10.761647 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:10.761760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.762345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:10.762345 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:10.762345 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:10.762345 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:10.762345 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:10.762345 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:10.762345 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.762345 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:10.763294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.763294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:10.763294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:10.763294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:10.763294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.763294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:10.763294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:10.763294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.763294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:10.763294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:10.763294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:10.763294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.763641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:10.763641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:10.763641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:10.763641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:10.763641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:10.763641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:10.763641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:10.763641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:10.763641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:10.763641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:10.763641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:10.763641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:10.763641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:10.763873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.763873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:10.763873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.763873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:10.763873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:10.763873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:10.763873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:10.763873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:10.763873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.763873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:10.763873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:10.763873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.763873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:10.763873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:10.763873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.764186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:10.764186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:10.764186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.764186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:10.764186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:10.764186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.764186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:10.764186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:10.764186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:10.764186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.764186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:10.764186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:10.764186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:10.764186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:10.764186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:10.764186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:10.764186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:10.764186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:10.764186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:10.764533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:10.764533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:10.764533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:10.764533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:10.764533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.764533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:10.781240 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:10.781585 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:10.781696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:10.781793 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:31:10.782153 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:10.782353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.782792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:10.782792 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:10.782792 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:10.782792 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:10.782792 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:10.782792 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:10.782792 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.782792 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:10.783498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.783498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:10.783498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:10.783498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:10.783498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.783498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:10.783498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:10.783498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:10.783498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.783498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:10.783498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:10.783498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:10.783498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:10.783498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:10.783498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:10.783783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:10.783783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:10.783783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:10.783783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:10.783783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:10.783783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:10.783783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:10.783783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.783783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:31:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.784161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:10.784405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:10.784405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:10.784405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.784405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:10.784405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:10.784405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:10.784405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:10.784405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:10.784405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:10.784405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:10.784405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:10.784405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:10.784405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:10.784405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:10.784632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:10.784632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:10.784632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.784632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:10.784892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.784892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:10.784892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:10.784892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:10.784892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:10.784892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:10.784892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.784892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:10.784892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:10.784892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.784892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:10.784892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:10.784892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.784892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:10.784892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:10.784892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.785356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:10.785356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:10.785356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:31:10.785356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:10.785356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:10.785356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.785356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:10.785356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:10.785356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:10.785356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.785356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:10.785356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:10.785356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:10.785356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:10.785356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:10.785356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:10.785356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:10.785356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:10.785678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:10.785678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:10.785678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:10.785678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:10.785678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:10.785678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.785678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:10.787918 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:10.788168 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:10.788294 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:10.788420 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:10.788835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:10.789064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.789539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:10.789539 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:10.789539 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:10.789539 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:10.789539 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:10.789539 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:10.789539 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.789539 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:10.790120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.790120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:10.790120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:10.790120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:10.790120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.790120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:10.790120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:10.790120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:10.790120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.790120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:10.790120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:10.790120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:10.790120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:10.790120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:10.790120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:10.790444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:10.790444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:10.790444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:10.790444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:10.790444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:10.790444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:10.790444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:10.790444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.790444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:10.790730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.790730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:10.790730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:10.790730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:10.790730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.790730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:10.790730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:10.790730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:10.790730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:10.790730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.790730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:10.790960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:10.790960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:10.790960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.790960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:10.790960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:10.790960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:10.790960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:10.790960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:10.790960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:10.790960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:10.790960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:10.790960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:10.790960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:10.790960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:10.791218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:10.791218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:10.791218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.791218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:10.791495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.791495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:10.791495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:10.791495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:10.791495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:10.791495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:10.791495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.791495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:10.791495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:10.791495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.791495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:10.791495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:10.791495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.791495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:10.791495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:10.791495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.791495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:10.791798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:10.791798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:10.791798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:10.791798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.791798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:10.791798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:10.791798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:10.791798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.791798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:10.791798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:10.791798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:10.791798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:10.791798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:10.791798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:10.791798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:10.791798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:10.791798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:10.791798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:10.791798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:10.792142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:10.792142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:10.792142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.792142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:10.793263 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:10.793485 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:10.793732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:10.793732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:10.794690 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:10.794925 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:10.795239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.795763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:10.795763 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:10.795763 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:10.795763 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:10.795763 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:10.795763 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:10.795763 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.795763 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:10.796462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.796462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:10.796462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:10.796462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:10.796462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.796462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:10.796462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:10.796462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.796462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:10.796462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:10.796462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:10.796462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.796682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:10.796682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:10.796682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:10.796682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:10.796682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:10.796682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:10.796682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:10.796682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:10.796682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:10.796682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:10.796682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:10.796682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:10.796682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:10.796924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.796924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:10.797217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.797217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:10.797217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:10.797217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:10.797217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:10.797217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:10.797217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.797217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:10.797217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:10.797217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.797217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:10.797217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:10.797217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.797217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:10.797217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:10.797217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:10.797818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:10.797818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:10.797818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.797818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:10.797818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:10.797818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:10.797818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:10.797818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:10.797818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:10.797818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:10.797818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:10.797818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:10.797818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:10.797818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:10.797818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:10.797818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:10.797818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:10.797818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:10.797818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:10.798532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:10.798532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:10.798532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:13.681933 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:13.683616 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:13.683721 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:13.684342 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:13.732835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:13.733059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:13.733632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:13.733632 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:13.733632 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:13.733632 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:13.733632 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:13.733632 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:13.733632 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:13.733632 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:13.734674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.050 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:13.734674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:13.734674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:13.734674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:13.734674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:13.734674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:13.734674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:13.734674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:13.734674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:13.734674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:13.734674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:13.734674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:13.734923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:13.734923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:13.734923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:13.734923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:13.734923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:13.734923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:13.734923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:13.734923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:13.734923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:13.734923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:13.734923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:13.734923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:13.734923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:13.735141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:13.735141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:13.736369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:13.736369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:13.736369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:13.736369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:13.736369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:13.736369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:13.736369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:13.736369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:13.736369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:13.736369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:13.736369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:13.736369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:13.736369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:13.736369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:13.736369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:13.736369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:13.736667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:13.736667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:13.736667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:13.736667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:13.736667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:13.736667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:13.736667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:13.736667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:13.736667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:13.736667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:13.736667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:13.736667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:13.736667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:13.736667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:13.736667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:13.736667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:13.736667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:13.736667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:13.736667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:13.737005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:13.737005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:13.737005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:15.759901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:15.760942 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:15.760942 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:15.761729 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:15.762134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:15.762576 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:15.762576 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:15.763079 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:15.763079 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:15.763617 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:15.763617 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:15.763617 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:15.763874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.764570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:15.764570 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:15.764570 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:15.764570 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:15.764570 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:15.764570 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:15.764570 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.764570 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:15.765839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.765839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:15.765839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:15.765839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:15.765839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.765839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:15.765839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:15.765839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.765839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:15.765839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:15.765839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:15.765839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.766279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:15.766279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:15.766279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:15.766279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:15.766279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:15.766279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:15.766279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:15.766279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:15.766279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:15.766279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:15.766279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:15.766279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:15.766279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:15.766690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.766690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:15.767851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.767851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:15.767851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:15.767851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:15.767851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:15.767851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:15.767851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.767851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:15.767851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:15.767851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.767851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:15.767851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:15.767851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.767851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:15.767851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:15.767851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.768385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:15.768385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:15.768385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.768385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:15.768385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:15.768385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:15.768385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.768385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:15.768385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:15.768385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:15.768385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:15.768385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:15.768385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:15.768385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:15.768385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:15.768385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:15.768385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:15.768385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:15.768385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:15.769026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:15.769026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.769026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:15.781709 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:15.783848 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:15.784031 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:15.784099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:31:15.784795 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:15.784795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.785335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:15.785335 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:15.785335 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:15.785335 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:15.785335 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:15.785335 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:15.785335 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.785335 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:15.787018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.787018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:15.787018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:15.787018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:15.787018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.787018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:15.787018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:15.787018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:15.787018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.787018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:15.787018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:15.787018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:15.787018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:15.787018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:15.787018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:15.787334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:15.787334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:15.787334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:15.787334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:15.787334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:15.787334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:15.787334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:15.787334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.787334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:15.788439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.788439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:15.788439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:15.788439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:15.788439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.788439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:15.788439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:15.788439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:31:15.788439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:15.788439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:15.788439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.788439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:15.788681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:15.788681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:15.788681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.788681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:15.788681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:15.788681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:15.788681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:15.788681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:15.788681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:15.788681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:15.788681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:15.788681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:15.788681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:15.788681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:15.788889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:15.788889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:15.788889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.788889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:15.790572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.790572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:15.790572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:15.790572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:15.790572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:15.790572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:15.790572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.790572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:15.790572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:15.790572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.790572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:15.790572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:15.790572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.790572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:15.790572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:15.790572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.790867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:15.790867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:15.790867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:31:15.790867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:15.790867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:15.790867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.790867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:15.790867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:15.790867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:15.790867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.790867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:15.790867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:15.790867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:15.790867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:15.790867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:15.790867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:15.790867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:15.790867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:15.791231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:15.791231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:15.791231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:15.791231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:15.791231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:15.791231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.791231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:15.791431 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:15.792072 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:15.792197 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:15.792333 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:15.792967 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:15.793405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.793704 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:15.793985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:15.793985 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:15.793985 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:15.793985 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:15.793985 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:15.793985 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:15.793985 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.793985 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:15.794834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.794834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:15.794834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:15.794834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:15.794834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.794834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:15.794834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:15.794834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:15.794834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.794834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:15.794834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:15.794834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:15.794834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:15.794834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:15.794834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:15.795125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:15.795125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:15.795125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:15.795125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:15.795125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:15.795125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:15.795125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:15.795125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.795125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:15.795686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.795686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:15.795686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:15.795686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:15.795686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.795686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:15.795686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:15.795686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:15.795686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:15.795686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.795686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:15.796735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:15.796735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:15.796735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.796735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:15.796735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:15.796735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:15.796735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:15.796735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:15.796735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:15.796735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:15.796735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:15.796735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:15.796735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:15.796735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:15.796941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:15.796941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:15.796941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.796941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:15.796941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.796941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:15.796941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:15.796941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:15.796941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:15.796941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:15.796941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.796941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:15.796941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:15.796941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.797200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:15.797200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:15.797200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.797200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:15.797200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:15.797200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.797200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:15.797200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:15.797200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:15.797200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:15.797200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.797200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:15.797200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:15.797200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:15.797200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.797200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:15.797200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:15.797200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:15.797200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:15.797503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:15.797503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:15.797503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:15.797503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:15.797503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:15.797503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:15.797503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:15.797503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:15.797503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:15.797503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.797503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:15.797503 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:15.797503 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:15.797692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:15.798500 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:15.798742 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:15.799621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.800434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:15.800434 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:15.800434 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:15.800434 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:15.800434 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:15.800434 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:15.800434 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.800434 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:15.801322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.801322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:15.801322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:15.801322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:15.801322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.801322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:15.801322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:15.801322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.801322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:15.801322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:15.801322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:15.801322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.801546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:15.801546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:15.801546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:15.801546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:15.801546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:15.801546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:15.801546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:15.801546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:15.801546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:15.801546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:15.801546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:15.801546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:15.801546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:15.801763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.801763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:15.804345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.804345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:15.804345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:15.804345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:15.804345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:15.804345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:15.804345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.804345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:15.804345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:15.804345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.804345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:15.804345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:15.804345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.804345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:15.804345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:15.804345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:15.804626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:15.804626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:15.804626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.804626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:15.804626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:15.804626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:15.804626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:15.804626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:15.804626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:15.804626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:15.804626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:15.804626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:15.804626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:15.804626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:15.804626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:15.804626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:15.804626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:15.804626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:15.804626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:15.804915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:15.804915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:15.804915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:18.683636 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:18.684350 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:18.684469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:18.685094 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:18.728776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:18.729031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:18.729636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:18.729636 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:18.729636 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:18.729636 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:18.729636 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:18.729636 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:18.729636 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:18.729636 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:18.730350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:18.730350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:18.730350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:18.730350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:18.730350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:18.730350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:18.730350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:18.730350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:18.730350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:18.730350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:18.730350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:18.730350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:18.730721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:18.730721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:18.730721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:18.730721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:18.730721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:18.730721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:18.730721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:18.730721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:18.730721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:18.730721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:18.730721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:18.730721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:18.730721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:18.731077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:18.731077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:18.731077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:18.731077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:18.731077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:18.731077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:18.731077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:18.731077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:18.731077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:18.731077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:18.731077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:18.731077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:18.731077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:18.731077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:18.731077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:18.731524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:18.731524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:18.731524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:18.731524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:18.731524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:18.731524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:18.731524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:18.731524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:18.731524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:18.731524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:18.731524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:18.731524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:18.731524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:18.731524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:18.731524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:18.731524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:18.731524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:18.731524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:18.731524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:18.732020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:18.732020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:18.732020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:18.732020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:18.732020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:18.732020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:20.760346 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:20.760906 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:20.760906 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:20.761661 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:20.761841 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:20.762026 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:20.762157 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:20.762317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:20.762450 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:20.762600 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:20.762726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbcb05ba-3761-4282-b616-71a4937f4fed, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:20.762866 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:20.763005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.763568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:20.763568 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:20.763568 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:20.763568 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:20.763568 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:20.763568 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:20.763568 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.763568 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:20.764458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.764458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:20.764458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:20.764458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:20.764458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.764458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:20.764458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:20.764458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.764458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:20.764458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:20.764458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:20.764458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.764813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:20.764813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:20.764813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:20.764813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:20.764813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:20.764813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:20.764813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:20.764813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:20.764813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:20.764813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:20.764813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:20.764813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:20.764813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:20.765154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.765154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:20.765216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.765216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:20.765216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:20.765216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:20.765216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:20.765216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:20.765216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.765216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:20.765216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:20.765216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.765216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:20.765216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:20.765216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.765216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:20.765216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:20.765216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.765615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:20.765615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:20.765615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.765615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:20.765615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:20.765615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:20.765615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.765615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:20.765615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:20.765615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:20.765615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:20.765615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:20.765615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:20.765615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:20.765615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:20.765615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:20.765615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:20.765615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:20.765615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:20.766159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:20.766159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.766159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:20.783698 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:20.785138 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:20.785138 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:20.785138 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:31:20.785514 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:20.785700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.788212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:20.788212 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:20.788212 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:20.788212 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:20.788212 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:20.788212 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:20.788212 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.788212 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:20.788212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.788212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:20.788212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:20.788212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:20.788212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.788212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:20.788212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:20.788212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:20.788212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.788212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:20.788723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:20.788723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:20.788723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:20.788723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:20.788723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:20.788723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:20.788723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:20.788723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:20.788723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:20.788723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:20.788723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:20.788723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:20.788723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.788723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:20.788723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.789066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:20.789066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:20.789066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:20.789066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.789066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:20.789066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:20.789066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:31:20.789066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:20.789066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:20.789066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.789066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:20.789066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:20.789066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:20.789366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.789366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:20.789366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:20.789366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:20.789366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:20.789366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:20.789366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:20.789366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:20.789366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:20.789366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:20.789366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:20.789366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:20.789366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:20.789366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:20.789700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.789700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:20.791462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.791462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:20.791462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:20.791462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:20.791462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:20.791462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:20.791462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.791462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:20.791462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:20.791462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.791462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:20.791462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:20.791462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.791462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:20.791462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:20.791462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.792300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:20.792300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:20.792300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:31:20.792300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:20.792300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:20.792300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.792300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:20.792300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:20.792300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:20.792300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.792300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:20.792300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:20.792300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:20.792300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:20.792300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:20.792300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:20.792300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:20.792300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:20.793126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:20.793126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:20.793126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:20.793126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:20.793126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:20.793126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.793126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:20.793640 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:20.794291 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:20.794291 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:20.794291 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:20.795495 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:20.795724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.797774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:20.797774 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:20.797774 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:20.797774 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:20.797774 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:20.797774 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:20.797774 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.797774 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:20.797774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.797774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:20.797774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:20.797774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:20.797774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.797774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:20.797774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:20.797774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:20.797774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.797774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:20.798119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:20.798119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:20.798119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:20.798119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:20.798119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:20.798119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:20.798119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:20.798119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:20.798119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:20.798119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:20.798119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:20.798119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:20.798119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.798119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:20.798119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.798441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:20.798441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:20.798441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:20.798441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.798441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:20.798441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:20.798441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:20.798441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:20.798441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.798441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:20.798441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:20.798441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:20.798441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.798643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:20.798643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:20.798643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:20.798643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:20.798643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:20.798643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:20.798643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:20.798643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:20.798643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:20.798643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:20.798643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:20.798643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:20.798643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:20.798835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.798835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:20.799748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.799748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:20.799748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:20.799748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:20.799748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:20.799748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:20.799748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.799748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:20.799748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:20.799748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.799748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:20.799748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:20.799748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.799748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:20.799748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:20.799748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.799748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:20.800032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:20.800032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:20.800032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:20.800032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.800032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:20.800032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:20.800032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:20.800032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.800032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:20.800032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:20.800032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:20.800032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:20.800032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:20.800032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:20.800032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:20.800032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:20.800032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:20.800032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:20.800032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:20.800399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:20.800399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:20.800399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.800399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:20.800796 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:20.801322 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:20.801702 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:20.801702 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:20.802010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:20.802279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:20.802481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.804142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:20.804142 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:20.804142 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:20.804142 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:20.804142 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:20.804142 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:20.804142 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.804142 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:20.804142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.804142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:20.804142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:20.804142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:20.804142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.804142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:20.804142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:20.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:20.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:20.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:20.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:20.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:20.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:20.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:20.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:20.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:20.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:20.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:20.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:20.804643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:20.804643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:20.804643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:20.804643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:20.804643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.804643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:20.805014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.805014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:20.805014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:20.805014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:20.805014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:20.805014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:20.805014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.805014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:20.805014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:20.805014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.805014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:20.805014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:20.805014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.805014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:20.805014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:20.805014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:20.805278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:20.805278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:20.805278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.805278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:20.805278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:20.805278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:20.805278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:20.805278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:20.805278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:20.805278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:20.805278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:20.805278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:20.805278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:20.805278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:20.805278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:20.805278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:20.805278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:20.805278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:20.805278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:20.805554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:20.805554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:20.805554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:23.684477 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:23.685596 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:23.685720 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:23.686504 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:23.746010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:23.746159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:23.748263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:23.748263 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:23.748263 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:23.748263 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:23.748263 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:23.748263 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:23.748263 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:23.748263 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:23.748263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.061 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:23.748263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:23.748263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:23.748263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:23.748263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:23.748263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:23.748263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:23.748717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:23.748717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:23.748717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:23.748717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:23.748717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:23.748717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:23.748717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:23.748717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:23.748717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:23.748717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:23.748717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:23.748717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:23.748717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:23.748717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:23.749120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:23.749120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:23.749120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:23.749120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:23.749120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:23.749120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:23.749120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:23.749120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:23.749120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:23.749120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:23.749120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:23.749120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:23.749120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:23.749120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:23.749519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:23.750091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:23.750091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:23.750091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:23.750091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:23.750091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:23.750091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:23.750091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:23.750091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:23.750091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:25.760811 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:25.761445 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:25.761445 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:25.762547 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:25.762833 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:25.763230 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:25.763441 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:25.763622 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:25.763812 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:25.764199 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:25.764399 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:25.764578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.765238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:25.765238 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:25.765238 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:25.765238 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:25.765238 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:25.765238 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:25.765238 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.765238 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:25.766101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.766101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:25.766101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:25.766101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:25.766101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.766101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:25.766101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:25.766101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.766101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:25.766101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:25.766101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:25.766101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.766524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:25.766524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:25.766524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:25.766524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:25.766524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:25.766524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:25.766524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:25.766524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:25.766524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:25.766524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:25.766524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:25.766524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:25.766524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:25.766892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.766892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:25.767737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.767737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:25.767737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:25.767737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:25.767737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:25.767737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:25.767737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.767737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:25.767737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:25.767737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.767737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:25.767737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:25.767737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.767737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:25.767737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:25.767737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.768265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:25.768265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:25.768265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.768265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:25.768265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:25.768265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:25.768265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.768265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:25.768265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:25.768265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:25.768265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:25.768265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:25.768265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:25.768265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:25.768265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:25.768265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:25.768265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:25.768265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:25.768265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:25.768823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:25.768823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.768823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:25.784904 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:25.785381 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:25.785546 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:25.785684 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:31:25.786005 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:25.786449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.786931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:25.786931 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:25.786931 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:25.786931 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:25.786931 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:25.786931 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:25.786931 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.786931 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:25.787896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.787896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:25.787896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:25.787896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:25.787896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.787896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:25.787896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:25.787896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:25.787896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.787896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:25.787896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:25.787896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:25.787896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:25.787896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:25.787896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:25.788413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:25.788413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:25.788413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:25.788413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:25.788413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:25.788413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:25.788413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:25.788413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.788413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:25.789490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.789490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:25.789490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:25.789490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:25.789490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.789490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:25.789490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:25.789490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:31:25.789490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:25.789490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:25.789490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.789490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:25.789892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:25.789892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:25.789892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.789892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:25.789892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:25.789892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:25.789892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:25.789892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:25.789892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:25.789892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:25.789892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:25.789892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:25.789892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:25.789892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:25.790298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:25.790298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:25.790298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.790298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:25.791682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.791682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:25.791682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:25.791682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:25.791682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:25.791682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:25.791682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.791682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:25.791682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:25.791682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.791682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:25.791682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:25.791682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.791682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:25.791682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:25.791682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.792377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:25.792377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:25.792377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:31:25.792377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:25.792377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:25.792377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.792377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:25.792377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:25.792377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:25.792377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.792377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:25.792377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:25.792377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:25.792377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:25.792377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:25.792377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:25.792377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:25.792377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:25.792906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:25.792906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:25.792906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:25.792906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:25.792906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:25.792906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.792906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:25.794148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:25.794446 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:25.794594 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:25.794726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:25.795074 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:25.795358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.795845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:25.795845 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:25.795845 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:25.795845 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:25.795845 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:25.795845 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:25.795845 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.795845 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:25.796725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.796725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:25.796725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:25.796725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:25.796725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.796725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:25.796725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:25.796725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:25.796725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.796725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:25.796725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:25.796725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:25.796725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:25.796725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:25.796725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:25.797204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:25.797204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:25.797204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:25.797204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:25.797204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:25.797204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:25.797204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:25.797204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.797204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:25.798103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.798103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:25.798103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:25.798103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:25.798103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.798103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:25.798103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:25.798103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:25.798103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:25.798103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.798103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:25.798567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:25.798567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:25.798567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.798567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:25.798567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:25.798567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:25.798567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:25.798567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:25.798567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:25.798567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:25.798567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:25.798567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:25.798567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:25.798567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:25.798889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:25.798889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:25.798889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.798889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:25.799513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.799513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:25.799513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:25.799513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:25.799513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:25.799513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:25.799513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.799513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:25.799513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:25.799513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.799513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:25.799513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:25.799513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.799513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:25.799513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:25.799513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.799513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:25.799982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:25.799982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:25.799982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:25.799982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.799982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:25.799982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:25.799982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:25.799982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.799982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:25.799982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:25.799982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:25.799982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:25.799982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:25.799982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:25.799982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:25.799982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:25.799982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:25.799982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:25.799982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:25.800494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:25.800494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:25.800494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.800494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:25.801480 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:25.801714 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:25.802326 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:25.802326 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:25.802559 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:25.802749 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:25.803060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.803552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:25.803552 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:25.803552 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:25.803552 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:25.803552 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:25.803552 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:25.803552 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.803552 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:25.804447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.804447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:25.804447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:25.804447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:25.804447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.804447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:25.804447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:25.804447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.804447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:25.804447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:25.804447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:25.804447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.804824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:25.804824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:25.804824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:25.804824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:25.804824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:25.804824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:25.804824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:25.804824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:25.804824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:25.804824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:25.804824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:25.804824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:25.804824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:25.805193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.805193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:25.805866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.805866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:25.805866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:25.805866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:25.805866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:25.805866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:25.805866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.805866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:25.805866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:25.805866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.805866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:25.805866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:25.805866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.805866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:25.805866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:25.805866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:25.806430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:25.806430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:25.806430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.806430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:25.806430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:25.806430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:25.806430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:25.806430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:25.806430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:25.806430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:25.806430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:25.806430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:25.806430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:25.806430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:25.806430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:25.806430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:25.806430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:25.806430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:25.806430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:25.806933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:25.806933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:25.806933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:28.685718 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:28.687220 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:28.687339 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:28.688120 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:28.731778 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:28.732018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:28.734077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:28.734077 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:28.734077 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:28.734077 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:28.734077 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:28.734077 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:28.734077 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:28.734077 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:28.734077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:28.734077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:28.734077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:28.734077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:28.734077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:28.734077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:28.734077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:28.734632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:28.734632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:28.734632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:28.734632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:28.734632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:28.734632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:28.734632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:28.734632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:28.734632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:28.734632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:28.734632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:28.734632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:28.734632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:28.734632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:28.735014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:28.735014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:28.735014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:28.735014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:28.735014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:28.735014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:28.735014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:28.735014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:28.735014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:28.735014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:28.735014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:28.735014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:28.735014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:28.735014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:28.735418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:28.735987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:28.735987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:28.735987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:28.735987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:28.735987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:28.735987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:28.735987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:28.735987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:28.735987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:30.761223 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:30.761642 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:30.761790 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:30.762793 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:30.762900 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:30.763087 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:30.763216 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:30.763369 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:30.763515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:30.763647 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:30.764288 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:30.764446 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:30.764599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.765169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:30.765169 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:30.765169 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:30.765169 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:30.765169 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:30.765169 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:30.765169 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.765169 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:30.765805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.765805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:30.765805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:30.765805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:30.765805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.765805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:30.765805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:30.765805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.765805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:30.765805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:30.765805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:30.765805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.766197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:30.766197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:30.766197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:30.766197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:30.766197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:30.766197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:30.766197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:30.766197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:30.766197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:30.766197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:30.766197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:30.766197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:30.766197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:30.766551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.766551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:30.766551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.766551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:30.766551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:30.766551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:30.766551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:30.766551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:30.766551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.766551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:30.766551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:30.766551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.766551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:30.766551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:30.766551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.767311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:30.767311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:30.767311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.767311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:30.767311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:30.767311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.767311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:30.767311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:30.767311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:30.767311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.767311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:30.767311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:30.767311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:30.767311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:30.767311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:30.767311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:30.767311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:30.767311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:30.767311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:30.768467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:30.768467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:30.768467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:30.768467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:30.768467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.768467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:30.785610 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:30.785964 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:30.786122 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:30.786185 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:31:30.786636 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:30.786698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.787134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:30.787134 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:30.787134 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:30.787134 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:30.787134 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:30.787134 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:30.787134 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.787134 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:30.787848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.787848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:30.787848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:30.787848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:30.787848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.787848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:30.787848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:30.787848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:30.787848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.787848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:30.787848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:30.787848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:30.787848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:30.787848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:30.787848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:30.788343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:30.788343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:30.788343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:30.788343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:30.788343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:30.788343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:30.788343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:30.788343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.788343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:30.788815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.788815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:30.788815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:30.788815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:30.788815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.788815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:30.788815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:30.788815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:31:30.788815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:30.788815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:30.788815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.788815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:30.789145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:30.789145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:30.789145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.789145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:30.789145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:30.789145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:30.789145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:30.789145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:30.789145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:30.789145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:30.789145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:30.789145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:30.789145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:30.789145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:30.789565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:30.789565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:30.789565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.789565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:30.789565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.789565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:30.789565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:30.789565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:30.789565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:30.789565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:30.789565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.789565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:30.789565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:30.789565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.790125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:30.790125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:30.790125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.790125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:30.790125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:30.790125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.790125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:30.790125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:30.790125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:31:30.790125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:30.790125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:30.790125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.790125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:30.790125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:30.790125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:30.790125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.790125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:30.790125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:30.790125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:30.791526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:30.791526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:30.791526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:30.791526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:30.791526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:30.791526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:30.791526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:30.791526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:30.791526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:30.791526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:30.791526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.791526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:30.794775 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:30.795680 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:30.795842 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:30.795994 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:30.796723 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:30.796909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.797397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:30.797397 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:30.797397 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:30.797397 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:30.797397 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:30.797397 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:30.797397 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.797397 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:30.798528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.798528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:30.798528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:30.798528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:30.798528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.798528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:30.798528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:30.798528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:30.798528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.798528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:30.798528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:30.798528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:30.798528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:30.798528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:30.798528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:30.799021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:30.799021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:30.799021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:30.799021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:30.799021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:30.799021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:30.799021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:30.799021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.799021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:30.800306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.800306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:30.800306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:30.800306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:30.800306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.800306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:30.800306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:30.800306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:30.800306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:30.800306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.800306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:30.800840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:30.800840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:30.800840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.800840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:30.800840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:30.800840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:30.800840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:30.800840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:30.800840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:30.800840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:30.800840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:30.800840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:30.800840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:30.800840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:30.801308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:30.801308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:30.801308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.801308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:30.803541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.803541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:30.803541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:30.803541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:30.803541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:30.803541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:30.803541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.803541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:30.803541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:30.803541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.803541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:30.803541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:30.803541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.803541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:30.803541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:30.803541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.803541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:30.804123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:30.804123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:30.804123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:30.804123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.804123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:30.804123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:30.804123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:30.804123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.804123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:30.804123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:30.804123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:30.804123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:30.804123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:30.804123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:30.804123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:30.804123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:30.804123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:30.804123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:30.804123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:30.804623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:30.804623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:30.804623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.804623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:30.804623 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:30.804623 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:30.804800 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:30.804838 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:30.806496 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:30.806706 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:30.806883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.807307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:30.807307 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:30.807307 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:30.807307 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:30.807307 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:30.807307 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:30.807307 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.807307 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:30.810355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.810355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:30.810355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:30.810355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:30.810355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.810355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:30.810355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:30.810355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.810355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:30.810355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:30.810355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:30.810355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.811041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:30.811041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:30.811041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:30.811041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:30.811041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:30.811041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:30.811041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:30.811041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:30.811041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:30.811041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:30.811041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:30.811041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:30.811041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:30.811621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.811621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:30.811621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.811621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:30.811621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:30.811621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:30.811621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:30.811621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:30.811621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.811621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:30.811621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:30.811621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.811621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:30.811621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:30.811621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.812012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:30.812012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:30.812012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:30.812012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:30.812012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:30.812012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.812012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:30.812012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:30.812012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:30.812012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:30.812012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:30.812012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:30.812012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:30.812012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:30.812012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:30.812012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:30.812012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:30.812012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:30.812012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:30.812572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:30.812572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:30.812572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:30.812572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:30.812572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:30.812572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:33.687306 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:33.688105 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:33.688219 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:33.688835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:33.720051 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:33.720178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:33.720588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:33.720588 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:33.720588 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:33.720588 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:33.720588 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:33.720588 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:33.720588 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:33.720588 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:33.721229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:33.721229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:33.721229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:33.721229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:33.721229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:33.721229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:33.721229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:33.721229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:33.721229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:33.721229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:33.721229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:33.721229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:33.721469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:33.721469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:33.721469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:33.721469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:33.721469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:33.721469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:33.721469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:33.721469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:33.721469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:33.721469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:33.721469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:33.721469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:33.721469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:33.721665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:33.721665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:33.722211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:33.722211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:33.722211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:33.722211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:33.722211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:33.722211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:33.722211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:33.722211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:33.722211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:33.722211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:33.722211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:33.722211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:33.722211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:33.722211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:33.722211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:33.722211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:33.722721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:33.722721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:33.722721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:33.722721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:33.722721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:33.722721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:33.722721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:33.722721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:33.722721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:33.722721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:33.722721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:33.722721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:33.722721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:33.722721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:33.722721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:33.722721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:33.722721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:33.722721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:33.722721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:33.723297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:33.723297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:33.723297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:35.761704 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:35.763330 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:35.763330 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:35.764224 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:35.764455 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:35.764650 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:35.764824 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:35.765013 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:35.765169 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:35.765346 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:35.765504 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-30bf0b43-acea-45e6-9fcd-f04d92d84af8, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:35.765655 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:35.765807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:35.765938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.767285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:35.767285 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:35.767285 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:35.767285 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:35.767285 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:35.767285 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:35.767285 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.767285 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:35.768151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.768151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:35.768151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:35.768151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:35.768151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.768151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:35.768151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:35.768151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.768151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:35.768151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:35.768151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:35.768151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.768533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:35.768533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:35.768533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:35.768533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:35.768533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:35.768533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:35.768533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:35.768533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:35.768533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:35.768533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:35.768533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:35.768533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:35.768533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:35.768855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.768855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:35.769242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.769242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:35.769242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:35.769242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:35.769242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:35.769242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:35.769242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.769242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:35.769242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:35.769242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.769242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:35.769242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:35.769242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.769242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:35.769242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:35.769242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.769944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:35.769944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:35.769944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.769944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:35.769944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:35.769944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:35.769944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.769944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:35.769944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:35.769944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:35.769944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:35.769944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:35.769944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:35.769944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:35.769944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:35.769944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:35.769944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:35.769944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:35.769944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:35.770461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:35.770461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.770461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:35.786617 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:35.786932 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:35.787114 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:35.787161 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:31:35.787768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:35.788019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.788685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:35.788685 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:35.788685 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:35.788685 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:35.788685 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:35.788685 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:35.788685 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.788685 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:35.789676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.789676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:35.789676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:35.789676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:35.789676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.789676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:35.789676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:35.789676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:35.789676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.789676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:35.789676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:35.789676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:35.789676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:35.789676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:35.789676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:35.790085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:35.790085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:35.790085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:35.790085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:35.790085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:35.790085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:35.790085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:35.790085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.790085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:35.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:35.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:35.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:35.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:35.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:35.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:31:35.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:35.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:35.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:35.790501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:35.790501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:35.790501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.790501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:35.790501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:35.790501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:35.790501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:35.790501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:35.790501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:35.790501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:35.790501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:35.790501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:35.790501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:35.790501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:35.790708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:35.790708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:35.790708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.790708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:35.791922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.791922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:35.791922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:35.791922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:35.791922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:35.791922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:35.791922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.791922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:35.791922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:35.791922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.791922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:35.791922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:35.791922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.791922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:35.791922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:35.791922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.793525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:35.793525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:35.793525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:31:35.793525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:35.793525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:35.793525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.793525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:35.793525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:35.793525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:35.793525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.793525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:35.793525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:35.793525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:35.793525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:35.793525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:35.793525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:35.793525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:35.793525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:35.794038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:35.794038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:35.794038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:35.794038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:35.794038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:35.794038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.794038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:35.795828 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:35.796016 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:35.796135 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:35.796303 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:35.797653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:35.797902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.798556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:35.798556 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:35.798556 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:35.798556 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:35.798556 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:35.798556 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:35.798556 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.798556 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:35.799083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.799083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:35.799083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:35.799083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:35.799083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.799083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:35.799083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:35.799083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:35.799083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.799083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:35.799083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:35.799083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:35.799083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:35.799083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:35.799083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:35.799700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:35.799700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:35.799700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:35.799700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:35.799700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:35.799700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:35.799700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:35.799700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.799700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:35.800052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.800052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:35.800052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:35.800052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:35.800052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.800052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:35.800052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:35.800052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:35.800052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:35.800052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.800052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:35.800359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:35.800359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:35.800359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.800359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:35.800359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:35.800359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:35.800359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:35.800359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:35.800359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:35.800359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:35.800359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:35.800359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:35.800359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:35.800359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:35.800659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:35.800659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:35.800659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.800659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:35.800893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.800893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:35.800893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:35.800893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:35.800893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:35.800893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:35.800893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.800893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:35.800893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:35.800893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.800893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:35.800893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:35.800893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.800893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:35.800893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:35.800893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.800893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:35.801309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:35.801309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:35.801309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:35.801309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.801309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:35.801309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:35.801309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:35.801309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.801309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:35.801309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:35.801309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:35.801309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:35.801309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:35.801309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:35.801309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:35.801309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:35.801309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:35.801309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:35.801309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:35.801665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:35.801665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:35.801665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.801665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:35.804869 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:35.805173 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:35.805320 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:35.805507 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:35.807681 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:35.807922 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:35.808228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.808851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:35.808851 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:35.808851 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:35.808851 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:35.808851 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:35.808851 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:35.808851 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.808851 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:35.809816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.809816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:35.809816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:35.809816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:35.809816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.809816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:35.809816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:35.809816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.809816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:35.809816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:35.809816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:35.809816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.810162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:35.810162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:35.810162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:35.810162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:35.810162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:35.810162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:35.810162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:35.810162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:35.810162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:35.810162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:35.810162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:35.810162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:35.810162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:35.810522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.810522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:35.810594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.810594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:35.810594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:35.810594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:35.810594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:35.810594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:35.810594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.810594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:35.810594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:35.810594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.810594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:35.810594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:35.810594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.810594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:35.810594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:35.810594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:35.811166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:35.811166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:35.811166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.811166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:35.811166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:35.811166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:35.811166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:35.811166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:35.811166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:35.811166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:35.811166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:35.811166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:35.811166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:35.811166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:35.811166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:35.811166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:35.811166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:35.811166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:35.811166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:35.811601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:35.811601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:35.811601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:38.688274 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:38.688843 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:38.688954 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:38.689571 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:38.738845 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:38.739097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:38.739633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:38.739633 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:38.739633 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:38.739633 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:38.739633 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:38.739633 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:38.739633 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:38.739633 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:38.740606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.051 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:38.740606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:38.740606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:38.740606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:38.740606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:38.740606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:38.740606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:38.740606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:38.740606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:38.740606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:38.740606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:38.740606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:38.740865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:38.740865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:38.740865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:38.740865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:38.740865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:38.740865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:38.740865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:38.740865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:38.740865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:38.740865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:38.740865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:38.740865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:38.740865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:38.741106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:38.741106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:38.741627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:38.741627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:38.741627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:38.741627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:38.741627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:38.741627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:38.741627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:38.741627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:38.741627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:38.741627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:38.741627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:38.741627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:38.741627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:38.741627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:38.741627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:38.741627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:38.742034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:38.742034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:38.742034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:38.742034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:38.742034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:38.742034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:38.742034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:38.742034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:38.742034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:38.742034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:38.742034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:38.742034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:38.742034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:38.742034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:38.742034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:38.742034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:38.742034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:38.742034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:38.742034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:38.742409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:38.742409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:38.742409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:40.762516 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:40.763434 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:40.763434 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:40.764424 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:40.764639 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:40.764808 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:40.765161 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:40.765161 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:40.765232 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:40.765352 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:40.765459 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:40.765556 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:40.765888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.766531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:40.766531 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:40.766531 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:40.766531 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:40.766531 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:40.766531 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:40.766531 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.766531 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:40.767469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.767469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:40.767469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:40.767469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:40.767469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.767469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:40.767469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:40.767469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.767469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:40.767469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:40.767469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:40.767469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.768014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:40.768014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:40.768014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:40.768014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:40.768014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:40.768014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:40.768014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:40.768014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:40.768014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:40.768014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:40.768014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:40.768014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:40.768014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:40.768712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.768712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:40.768754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.768754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:40.768754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:40.768754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:40.768754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:40.768754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:40.768754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.768754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:40.768754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:40.768754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.768754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:40.768754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:40.768754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.768754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:40.768754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:40.768754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.769028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:40.769028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:40.769028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.769028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:40.769028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:40.769028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:40.769028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.769028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:40.769028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:40.769028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:40.769028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:40.769028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:40.769028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:40.769028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:40.769028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:40.769028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:40.769028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:40.769028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:40.769028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:40.769362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:40.769362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.769362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:40.788324 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:40.788324 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:40.788324 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:40.788324 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:31:40.788324 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:40.788324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.788633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:40.788633 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:40.788633 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:40.788633 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:40.788633 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:40.788633 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:40.788633 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.788633 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:40.789243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.789243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:40.789243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:40.789243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:40.789243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.789243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:40.789243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:40.789243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:40.789243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.789243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:40.789243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:40.789243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:40.789243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:40.789243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:40.789243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:40.789530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:40.789530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:40.789530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:40.789530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:40.789530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:40.789530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:40.789530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:40.789530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.789530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:40.790027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.790027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:40.790027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:40.790027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:40.790027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.790027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:40.790027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:40.790027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:31:40.790027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:40.790027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:40.790027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.790027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:40.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:40.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:40.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:40.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:40.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:40.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:40.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:40.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:40.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:40.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:40.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:40.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:40.790305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:40.790515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:40.790515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:40.790515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.790515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:40.790891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.790891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:40.790891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:40.790891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:40.790891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:40.790891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:40.790891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.790891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:40.790891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:40.790891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.790891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:40.790891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:40.790891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.790891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:40.790891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:40.790891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.791223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:40.791223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:40.791223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:31:40.791223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:40.791223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:40.791223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.791223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:40.791223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:40.791223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:40.791223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.791223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:40.791223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:40.791223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:40.791223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:40.791223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:40.791223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:40.791223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:40.791223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:40.791579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:40.791579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:40.791579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:40.791579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:40.791579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:40.791579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.791579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:40.796048 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:40.796403 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:40.796553 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:40.796717 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:40.797207 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:40.797502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.798093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:40.798093 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:40.798093 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:40.798093 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:40.798093 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:40.798093 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:40.798093 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.798093 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:40.798902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.798902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:40.798902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:40.798902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:40.798902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.798902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:40.798902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:40.798902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:40.798902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.798902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:40.798902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:40.798902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:40.798902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:40.798902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:40.798902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:40.799150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:40.799150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:40.799150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:40.799150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:40.799150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:40.799150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:40.799150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:40.799150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.799150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:40.799697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.799697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:40.799697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:40.799697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:40.799697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.799697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:40.799697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:40.799697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:40.799697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:40.799697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.799697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:40.799929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:40.799929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:40.799929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.799929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:40.799929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:40.799929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:40.799929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:40.799929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:40.799929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:40.799929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:40.799929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:40.799929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:40.799929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:40.799929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:40.800159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:40.800159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:40.800159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.800159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:40.800707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.800707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:40.800707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:40.800707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:40.800707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:40.800707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:40.800707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.800707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:40.800707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:40.800707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.800707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:40.800707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:40.800707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.800707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:40.800707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:40.800707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.800707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:40.801070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:40.801070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:40.801070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:40.801070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.801070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:40.801070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:40.801070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:40.801070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.801070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:40.801070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:40.801070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:40.801070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:40.801070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:40.801070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:40.801070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:40.801070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:40.801070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:40.801070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:40.801070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:40.801399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:40.801399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:40.801399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.801399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:40.806268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:40.806268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:40.806268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:40.806268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:40.806268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:40.806476 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:40.806749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.807238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:40.807238 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:40.807238 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:40.807238 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:40.807238 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:40.807238 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:40.807238 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.807238 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:40.808196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.808196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:40.808196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:40.808196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:40.808196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.808196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:40.808196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:40.808196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.808196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:40.808196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:40.808196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:40.808196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.808501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:40.808501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:40.808501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:40.808501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:40.808501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:40.808501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:40.808501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:40.808501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:40.808501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:40.808501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:40.808501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:40.808501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:40.808501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:40.808861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.808861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:40.809173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.809173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:40.809173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:40.809173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:40.809173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:40.809173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:40.809173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.809173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:40.809173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:40.809173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.809173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:40.809173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:40.809173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.809173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:40.809173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:40.809173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:40.809552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:40.809552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:40.809552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.809552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:40.809552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:40.809552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:40.809552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:40.809552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:40.809552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:40.809552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:40.809552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:40.809552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:40.809552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:40.809552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:40.809552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:40.809552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:40.809552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:40.809552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:40.809552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:40.809950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:40.809950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:40.809950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:43.693285 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:43.693988 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:43.694353 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:43.695072 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:43.734412 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:43.734801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:43.735640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:43.735640 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:43.735640 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:43.735640 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:43.735640 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:43.735640 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:43.735640 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:43.735640 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:43.736676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:43.736676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:43.736676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:43.736676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:43.736676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:43.736676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:43.736676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:43.736676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:43.736676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:43.736676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:43.736676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:43.736676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:43.736966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:43.736966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:43.736966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:43.736966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:43.736966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:43.736966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:43.736966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:43.736966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:43.736966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:43.736966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:43.736966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:43.736966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:43.736966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:43.737304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:43.737304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:43.738064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:43.738064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:43.738064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:43.738064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:43.738064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:43.738064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:43.738064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:43.738064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:43.738064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:43.738064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:43.738064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:43.738064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:43.738064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:43.738064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:43.738064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:43.738064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:43.738470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:43.738470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:43.738470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:43.738470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:43.738470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:43.738470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:43.738470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:43.738470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:43.738470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:43.738470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:43.738470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:43.738470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:43.738470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:43.738470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:43.738470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:43.738470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:43.738470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:43.738470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:43.738470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:43.738849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:43.738849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:43.738849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:45.763498 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:45.763959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:45.763959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:45.764840 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:45.764994 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:45.765117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:45.765265 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:45.765358 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:45.765454 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:45.765563 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:45.765659 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:45.765758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:45.765859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.766643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:45.766643 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:45.766643 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:45.766643 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:45.766643 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:45.766643 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:45.766643 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.766643 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:45.767294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.767294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:45.767294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:45.767294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:45.767294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.767294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:45.767294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:45.767294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.767294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:45.767294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:45.767294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:45.767294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.767564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:45.767564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:45.767564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:45.767564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:45.767564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:45.767564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:45.767564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:45.767564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:45.767564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:45.767564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:45.767564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:45.767564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:45.767564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:45.767775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.767775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:45.767962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.767962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:45.767962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:45.767962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:45.767962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:45.767962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:45.767962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.767962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:45.767962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:45.767962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.767962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:45.767962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:45.767962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.767962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:45.767962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:45.767962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.768241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:45.768241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:45.768241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.768241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:45.768241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:45.768241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:45.768241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.768241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:45.768241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:45.768241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:45.768241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:45.768241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:45.768241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:45.768241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:45.768241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:45.768241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:45.768241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:45.768241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:45.768241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:45.768556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:45.768556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.768556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:45.787430 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:45.787684 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:45.787771 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:45.787865 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:31:45.788171 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:45.788348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.788868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:45.788868 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:45.788868 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:45.788868 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:45.788868 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:45.788868 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:45.788868 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.788868 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:45.789482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.789482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:45.789482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:45.789482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:45.789482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.789482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:45.789482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:45.789482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:45.789482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.789482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:45.789482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:45.789482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:45.789482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:45.789482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:45.789482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:45.789766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:45.789766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:45.789766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:45.789766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:45.789766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:45.789766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:45.789766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:45.789766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.789766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:45.790084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.790084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:45.790084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:45.790084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:45.790084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.790084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:45.790084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:45.790084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:31:45.790084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:45.790084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:45.790084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.790084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:45.790319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:45.790319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:45.790319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.790319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:45.790319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:45.790319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:45.790319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:45.790319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:45.790319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:45.790319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:45.790319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:45.790319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:45.790319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:45.790319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:45.790543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:45.790543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:45.790543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.790543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:45.790543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.790543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:45.790543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:45.790543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:45.790543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:45.790543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:45.790543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.790543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:45.790543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:45.790543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.790862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:45.790862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:45.790862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.790862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:45.790862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:45.790862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.790862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:45.790862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:45.790862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:31:45.790862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:45.790862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:45.790862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.790862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:45.790862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:45.790862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:45.790862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.790862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:45.790862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:45.790862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:45.791177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:45.791177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:45.791177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:45.791177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:45.791177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:45.791177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:45.791177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:45.791177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:45.791177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:45.791177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:45.791177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.791177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:45.796443 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:45.798579 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:45.798579 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:45.798579 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:45.798579 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:45.798579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.798579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:45.798579 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:45.798579 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:45.798579 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:45.798877 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:45.798877 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:45.798877 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.798877 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:45.798877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.798877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:45.798877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:45.798877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:45.798877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.798877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:45.798877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:45.798877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:45.798877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.798877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:45.798877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:45.798877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:45.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:45.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:45.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:45.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:45.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:45.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:45.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:45.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:45.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:45.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:45.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:45.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:45.799376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:45.799376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:45.799376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.799376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:45.799376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:45.799376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:45.799376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:45.799376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.799376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:45.799376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:45.799376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:45.799376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.799376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:45.799593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:45.799593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:45.799593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:45.799593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:45.799593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:45.799593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:45.799593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:45.799593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:45.799593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:45.799593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:45.799593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:45.799593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:45.799593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.799792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:45.799792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.799792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:45.799792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:45.799792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:45.799792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:45.799792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:45.799792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.799792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:45.799792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:45.799792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.799792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:45.799792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:45.799792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.799792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:45.799792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:45.799792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.800225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:45.800225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:45.800225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:45.800225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:45.800225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.800225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:45.800225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:45.800225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:45.800225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.800225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:45.800225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:45.800225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:45.800225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:45.800225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:45.800225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:45.800225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:45.800225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:45.800225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:45.800225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:45.800549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:45.800549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:45.800549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:45.800549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.800549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:45.805519 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:45.805703 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:45.805812 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:45.805936 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:45.806182 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:45.806388 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:45.806597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.806957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:45.806957 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:45.806957 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:45.806957 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:45.806957 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:45.806957 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:45.806957 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.806957 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:45.807552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.807552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:45.807552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:45.807552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:45.807552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.807552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:45.807552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:45.807552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.807552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:45.807552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:45.807552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:45.807552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.807802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:45.807802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:45.807802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:45.807802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:45.807802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:45.807802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:45.807802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:45.807802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:45.807802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:45.807802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:45.807802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:45.807802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:45.807802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:45.808042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.808042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:45.808231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.808231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:45.808231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:45.808231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:45.808231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:45.808231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:45.808231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.808231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:45.808231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:45.808231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.808231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:45.808231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:45.808231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.808231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:45.808231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:45.808231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:45.808543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:45.808543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:45.808543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.808543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:45.808543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:45.808543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:45.808543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:45.808543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:45.808543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:45.808543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:45.808543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:45.808543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:45.808543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:45.808543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:45.808543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:45.808543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:45.808543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:45.808543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:45.808543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:45.808858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:45.808858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:45.808858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:48.693708 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:48.695534 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:48.695534 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:48.695534 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:48.728961 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:48.729187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:48.731409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:48.731409 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:48.731409 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:48.731409 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:48.731409 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:48.731409 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:48.731409 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:48.731409 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:48.731409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:48.731409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:48.731409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:48.731409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:48.731409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:48.731409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:48.731409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:48.731721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:48.731721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:48.731721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:48.731721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:48.731721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:48.731721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:48.731721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:48.731721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:48.731721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:48.731721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:48.731721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:48.731721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:48.731721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:48.731721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:48.731948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:48.731948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:48.731948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:48.731948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:48.731948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:48.731948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:48.731948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:48.731948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:48.731948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:48.731948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:48.731948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:48.731948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:48.731948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:48.731948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:48.732199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:48.732648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:48.732648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:48.732648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:48.732648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:48.732648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:48.732648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:48.732648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:48.732648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:48.732648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:50.763853 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:50.764831 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:50.764926 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:50.767598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:50.767914 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:50.768111 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:50.768287 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:50.768454 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:50.768618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:50.768755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:50.768894 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:50.769041 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:50.769194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.770561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:50.770561 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:50.770561 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:50.770561 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:50.770561 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:50.770561 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:50.770561 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.770561 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:50.771181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.771181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:50.771181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:50.771181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:50.771181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.771181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:50.771181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:50.771181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.771181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:50.771181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:50.771181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:50.771181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.771424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:50.771424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:50.771424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:50.771424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:50.771424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:50.771424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:50.771424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:50.771424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:50.771424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:50.771424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:50.771424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:50.771424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:50.771424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:50.771610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.771610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:50.771946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.771946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:50.771946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:50.771946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:50.771946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:50.771946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:50.771946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.771946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:50.771946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:50.771946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.771946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:50.771946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:50.771946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.771946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:50.771946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:50.771946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.772193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:50.772193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:50.772193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.772193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:50.772193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:50.772193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:50.772193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.772193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:50.772193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:50.772193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:50.772193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:50.772193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:50.772193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:50.772193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:50.772193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:50.772193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:50.772193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:50.772193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:50.772193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:50.772521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:50.772521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.772521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:50.789426 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:50.789826 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:50.789937 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:50.790065 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:31:50.790449 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:50.790657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.791274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:50.791274 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:50.791274 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:50.791274 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:50.791274 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:50.791274 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:50.791274 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.791274 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:50.792074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.792074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:50.792074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:50.792074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:50.792074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.792074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:50.792074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:50.792074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:50.792074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.792074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:50.792074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:50.792074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:50.792074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:50.792074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:50.792074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:50.792503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:50.792503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:50.792503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:50.792503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:50.792503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:50.792503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:50.792503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:50.792503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.792503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:50.792890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.792890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:50.792890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:50.792890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:50.792890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.792890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:50.792890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:50.792890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:31:50.792890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:50.792890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:50.792890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.792890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:50.793238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:50.793238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:50.793238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.793238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:50.793238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:50.793238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:50.793238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:50.793238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:50.793238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:50.793238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:50.793238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:50.793238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:50.793238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:50.793238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:50.793613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:50.793613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:50.793613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.793613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:50.793808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.793808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:50.793808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:50.793808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:50.793808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:50.793808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:50.793808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.793808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:50.793808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:50.793808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.793808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:50.793808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:50.793808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.793808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:50.793808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:50.793808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.794232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:50.794232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:50.794232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:31:50.794232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:50.794232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:50.794232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.794232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:50.794232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:50.794232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:50.794232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.794232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:50.794232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:50.794232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:50.794232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:50.794232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:50.794232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:50.794232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:50.794232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:50.794701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:50.794701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:50.794701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:50.794701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:50.794701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:50.794701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.794701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:50.796774 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:50.797068 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:50.797117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:50.797185 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:50.797474 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:50.797654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.798021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:50.798021 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:50.798021 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:50.798021 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:50.798021 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:50.798021 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:50.798021 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.798021 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:50.798508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.798508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:50.798508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:50.798508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:50.798508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.798508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:50.798508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:50.798508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:50.798508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.798508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:50.798508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:50.798508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:50.798508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:50.798508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:50.798508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:50.798732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:50.798732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:50.798732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:50.798732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:50.798732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:50.798732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:50.798732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:50.798732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.798732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:50.799120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.799120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:50.799120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:50.799120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:50.799120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.799120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:50.799120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:50.799120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:50.799120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:50.799120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.799120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:50.799327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:50.799327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:50.799327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.799327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:50.799327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:50.799327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:50.799327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:50.799327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:50.799327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:50.799327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:50.799327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:50.799327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:50.799327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:50.799327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:50.799533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:50.799533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:50.799533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.799533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:50.799827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.799827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:50.799827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:50.799827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:50.799827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:50.799827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:50.799827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.799827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:50.799827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:50.799827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.799827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:50.799827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:50.799827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.799827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:50.799827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:50.799827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.799827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:50.800130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:50.800130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:50.800130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:50.800130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.800130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:50.800130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:50.800130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:50.800130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.800130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:50.800130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:50.800130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:50.800130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:50.800130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:50.800130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:50.800130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:50.800130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:50.800130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:50.800130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:50.800130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:50.800426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:50.800426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:50.800426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.800426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:50.805960 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:50.806294 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:50.806391 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:50.806520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:50.806781 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:50.807008 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:50.807234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.807619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:50.807619 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:50.807619 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:50.807619 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:50.807619 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:50.807619 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:50.807619 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.807619 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:50.808202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.808202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:50.808202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:50.808202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:50.808202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.808202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:50.808202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:50.808202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.808202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:50.808202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:50.808202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:50.808202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.808442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:50.808442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:50.808442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:50.808442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:50.808442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:50.808442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:50.808442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:50.808442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:50.808442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:50.808442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:50.808442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:50.808442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:50.808442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:50.808664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.808664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:50.808862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.808862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:50.808862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:50.808862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:50.808862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:50.808862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:50.808862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.808862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:50.808862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:50.808862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.808862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:50.808862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:50.808862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.808862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:50.808862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:50.808862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:50.809129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:50.809129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:50.809129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.809129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:50.809129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:50.809129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:50.809129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:50.809129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:50.809129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:50.809129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:50.809129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:50.809129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:50.809129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:50.809129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:50.809129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:50.809129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:50.809129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:50.809129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:50.809129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:50.809438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:50.809438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:50.809438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:53.694444 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:53.695118 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:53.695238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:53.696874 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:53.699376 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=1118,neutron:UPDATE=5,neutron:DELETE=20 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:31:53.738151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:53.738474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:53.739146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:53.739146 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:53.739146 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:53.739146 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:53.739146 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:53.739146 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:53.739146 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:53.739146 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:53.740067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:53.740067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:53.740067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:53.740067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:53.740067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:53.740067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:53.740067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:53.740067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:53.740067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:53.740067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:53.740067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:53.740067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:53.740659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:53.740659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:53.740659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:53.740659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:53.740659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:53.740659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:53.740659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:53.740659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:53.740659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:53.740659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:53.740659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:53.740659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:53.740659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:53.741062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:53.741062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:53.741489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:53.741489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:53.741489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:53.741489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:53.741489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:53.741489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:53.741489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:53.741489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:53.741489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:53.741489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:53.741489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:53.741489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:53.741489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:53.741489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:53.741489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:53.741489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:53.742024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:53.742024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:53.742024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:53.742024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:53.742024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:53.742024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:53.742024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:53.742024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:53.742024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:53.742024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:53.742024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:53.742024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:53.742024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:53.742024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:53.742024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:53.742024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:53.742024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:53.742024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:53.742024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:53.742564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:53.742564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:53.742564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:55.764892 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:55.765423 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:55.765476 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:55.766995 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:55.767261 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:55.767453 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:55.767850 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:55.768078 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:55.768225 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:55.768432 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:55.769053 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:55.769223 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:55.769436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.770486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:55.770486 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:55.770486 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:55.770486 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:55.770486 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:55.770486 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:55.770486 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.770486 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:55.772049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.772049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:55.772049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:55.772049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:55.772049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.772049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:55.772049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:55.772049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.772049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:55.772049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:55.772049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:55.772049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.772782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:55.772782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:55.772782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:55.772782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:55.772782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:55.772782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:55.772782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:55.772782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:55.772782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:55.772782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:55.772782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:55.772782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:55.772782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:55.773107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.773107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:55.773480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.773480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:55.773480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:55.773480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:55.773480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:55.773480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:55.773480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.773480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:55.773480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:55.773480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.773480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:55.773480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:55.773480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.773480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:55.773480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:55.773480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.773785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:31:55.773785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:55.773785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.773785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:55.773785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:55.773785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:55.773785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.773785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:55.773785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:55.773785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:55.773785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:55.773785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:55.773785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:55.773785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:55.773785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:55.773785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:55.773785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:55.773785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:55.773785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:55.774119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:55.774119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.774119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:55.789890 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:55.791226 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:55.791432 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:55.792754 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:31:55.797244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:55.798542 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:55.800243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.802836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:55.802836 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:55.802836 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:55.802836 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:55.802836 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:55.802836 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:55.802836 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.802836 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:55.803535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.803535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:55.803535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:55.803535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:55.803535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.803535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:55.803535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:55.803535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:55.803535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.803535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:55.803535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:55.803535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:55.803535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:55.803535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:55.803535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:55.803865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:55.803865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:55.803865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:55.803865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:55.803865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:55.803865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:55.803865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:55.803865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.803865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:55.804446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.012 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.804446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:55.804446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:55.804446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:55.804446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.804446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:55.804446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:55.804446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:31:55.804446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:55.804446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:55.804446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.804446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:55.805037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:55.805037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:55.805037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.805037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:55.805037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:55.805037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:55.805037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:55.805037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:55.805037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:55.805037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:55.805037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:55.805037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:55.805037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:55.805037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:55.805302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:55.805302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:55.805302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.805302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:55.805904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.805904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:55.805904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:55.805904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:55.805904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:55.805904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:55.805904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.805904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:55.805904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:55.805904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.805904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:55.805904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:55.805904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.805904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:55.805904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:55.805904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.806264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:31:55.806264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:31:55.806264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:31:55.806264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:55.806264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:55.806264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.806264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:55.806264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:55.806264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:55.806264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.806264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:55.806264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:55.806264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:55.806264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:55.806264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:55.806264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:55.806264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:55.806264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:55.806749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:55.806749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:55.806749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:55.806749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:55.806749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:55.806749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.806749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:55.806749 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:55.806749 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:55.806749 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:55.807116 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:55.807235 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:55.807477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.807880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:55.807880 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:55.807880 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:55.807880 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:55.807880 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:55.807880 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:55.807880 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.807880 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:55.808489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.808489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:31:55.808489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:55.808489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:55.808489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.808489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:55.808489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:55.808489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:31:55.808489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.808489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:31:55.808489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:55.808489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:31:55.808489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:55.808489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:31:55.808489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:31:55.808755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:55.808755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:31:55.808755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:55.808755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:31:55.808755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:31:55.808755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:55.808755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:31:55.808755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.808755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:31:55.809343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.809343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:55.809343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:55.809343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:55.809343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.809343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:55.809343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:55.809343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:55.809343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:55.809343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.809343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:55.809621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:55.809621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:55.809621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.809621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:55.809621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:55.809621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:55.809621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:55.809621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:55.809621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:55.809621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:55.809621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:55.809621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:55.809621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:55.809621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:55.809900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:55.809900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:55.809900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.809900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:55.810540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.810540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:55.810540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:55.810540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:55.810540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:55.810540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:55.810540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.810540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:55.810540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:55.810540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.810540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:55.810540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:55.810540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.810540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:55.810540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:55.810540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.810540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:31:55.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:31:55.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:31:55.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:31:55.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:55.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:55.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:55.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:55.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:55.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:55.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:55.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:55.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:55.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:55.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:55.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:55.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:55.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:55.811366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:55.811366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:55.811366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.811366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:55.811366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:55.811366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:55.811366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:55.811366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:55.811577 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:55.811833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.812266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:55.812266 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:55.812266 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:55.812266 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:55.812266 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:55.812266 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:55.812266 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.812266 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:55.812889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.812889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:55.812889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:55.812889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:55.812889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.812889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:55.812889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:55.812889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.812889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:55.812889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:55.812889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:55.812889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.813132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:55.813132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:55.813132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:55.813132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:55.813132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:55.813132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:55.813132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:55.813132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:55.813132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:55.813132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:55.813132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:55.813132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:55.813132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:55.813440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.813440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:55.813901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.813901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:55.813901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:55.813901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:55.813901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:55.813901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:55.813901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.813901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:55.813901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:55.813901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.813901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:55.813901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:55.813901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.813901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:55.813901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:55.813901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:55.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:31:55.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:55.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:55.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:55.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:55.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:55.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:55.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:55.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:55.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:55.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:55.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:55.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:55.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:55.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:55.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:55.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:55.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:55.814579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:55.814579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:55.814579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:31:58.695325 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:31:58.697593 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:31:58.697697 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:31:58.698329 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:31:58.736087 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:31:58.736330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:58.736968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:31:58.736968 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:58.736968 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:31:58.736968 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:31:58.736968 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:58.736968 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:31:58.736968 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:58.736968 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:31:58.737585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:58.737585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:31:58.737585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:58.737585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:31:58.737585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:58.737585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:58.737585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:58.737585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:58.737585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:58.737585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:58.737585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:31:58.737585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:58.738015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:31:58.738015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:58.738015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:31:58.738015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:58.738015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:31:58.738015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:31:58.738015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:58.738015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:31:58.738015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:58.738015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:31:58.738015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:31:58.738015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:58.738015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:31:58.738405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:58.738405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:31:58.738405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:58.738405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:31:58.738405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:31:58.738405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:31:58.738405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:31:58.738405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:31:58.738405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:58.738405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:31:58.738405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:31:58.738405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:58.738405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:31:58.738405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:31:58.738405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:31:58.738787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:31:58.738787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:31:58.738787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:31:58.738787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:31:58.738787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:31:58.738787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:58.738787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:58.738787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:31:58.738787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:31:58.738787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:31:58.738787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:31:58.738787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:31:58.738787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:31:58.738787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:31:58.738787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:31:58.738787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:31:58.738787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:31:58.738787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:31:58.738787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:31:58.739271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:31:58.739271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:31:58.739271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:31:58.739271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:31:58.739271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:31:58.739271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:00.765520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:00.765931 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:00.765931 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:00.768469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:00.768469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:00.768469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:00.768469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:00.768469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:00.768697 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:00.768697 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:00.768697 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:00.768697 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb66c1ba-ca61-4eca-a52b-e1bd5aacc770, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:00.768697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.768697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:00.768697 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:00.768697 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:00.768697 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:00.769021 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:00.769021 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:00.769021 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.769021 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:00.769523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.769523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:00.769523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:00.769523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:00.769523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.769523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:00.769523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:00.769523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.769523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:00.769523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:00.769523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:00.769523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.769887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:00.769887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:00.769887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:00.769887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:00.769887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:00.769887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:00.769887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:00.769887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:00.769887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:00.769887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:00.769887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:00.769887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:00.769887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:00.770290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.770290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:00.770612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.770612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:00.770612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:00.770612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:00.770612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:00.770612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:00.770612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.770612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:00.770612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:00.770612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.770612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:00.770612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:00.770612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.770612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:00.770612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:00.770612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.771035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:00.771035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:00.771035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.771035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:00.771035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:00.771035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:00.771035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.771035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:00.771035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:00.771035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:00.771035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:00.771035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:00.771035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:00.771035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:00.771035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:00.771035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:00.771035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:00.771035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:00.771035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:00.771539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:00.771539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.771539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:00.791202 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:00.791504 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:00.791620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:00.791748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:32:00.792656 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:00.792656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.792776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:00.792776 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:00.792776 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:00.792776 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:00.792776 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:00.792776 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:00.792776 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.792776 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:00.793575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.793575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:00.793575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:00.793575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:00.793575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.793575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:00.793575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:00.793575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:00.793575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.793575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:00.793575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:00.793575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:00.793575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:00.793575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:00.793575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:00.793999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:00.793999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:00.793999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:00.793999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:00.793999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:00.793999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:00.793999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:00.793999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.793999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:00.794269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.794269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:00.794269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:00.794269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:00.794269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.794269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:00.794269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:00.794269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:32:00.794269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:00.794269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:00.794269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.794269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:00.794556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:00.794556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:00.794556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.794556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:00.794556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:00.794556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:00.794556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:00.794556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:00.794556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:00.794556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:00.794556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:00.794556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:00.794556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:00.794556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:00.794892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:00.794892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:00.794892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.794892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:00.794892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.794892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:00.794892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:00.794892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:00.794892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:00.794892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:00.794892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.794892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:00.794892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:00.794892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.795247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:00.795247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:00.795247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.795247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:00.795247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:00.795247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.795247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:00.795247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:00.795247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:32:00.795247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:00.795247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:00.795247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.795247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:00.795247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:00.795247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:00.795247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.795247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:00.795247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:00.795247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:00.795733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:00.795733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:00.795733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:00.795733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:00.795733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:00.795733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:00.795733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:00.795733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:00.795733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:00.795733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:00.795733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.795733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:00.806329 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:00.806612 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:00.806734 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:00.806879 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:00.807871 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:00.807871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.807953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:00.807953 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:00.807953 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:00.807953 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:00.807953 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:00.807953 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:00.807953 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.807953 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:00.808486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.808486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:00.808486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:00.808486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:00.808486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.808486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:00.808486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:00.808486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:00.808486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.808486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:00.808486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:00.808486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:00.808486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:00.808486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:00.808486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:00.808878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:00.808878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:00.808878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:00.808878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:00.808878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:00.808878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:00.808878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:00.808878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.808878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:00.809128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.809128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:00.809128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:00.809128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:00.809128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.809128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:00.809128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:00.809128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:00.809128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:00.809128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.809128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:00.809416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:00.809416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:00.809416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.809416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:00.809416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:00.809416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:00.809416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:00.809416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:00.809416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:00.809416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:00.809416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:00.809416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:00.809416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:00.809416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:00.809842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:00.809842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:00.809842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.809842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:00.810196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.810196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:00.810196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:00.810196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:00.810196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:00.810196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:00.810196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.810196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:00.810196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:00.810196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.810196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:00.810196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:00.810196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.810196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:00.810196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:00.810196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.810196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:00.810641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:00.810641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:00.810641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:00.810641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.810641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:00.810641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:00.810641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:00.810641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.810641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:00.810641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:00.810641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:00.810641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:00.810641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:00.810641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:00.810641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:00.810641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:00.810641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:00.810641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:00.810641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:00.811080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:00.811080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:00.811080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.811080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:00.811080 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:00.811080 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:00.811239 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:00.811294 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:00.812454 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:00.812454 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:00.812454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.812642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:00.812642 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:00.812642 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:00.812642 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:00.812642 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:00.812642 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:00.812642 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.812642 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:00.813370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.813370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:00.813370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:00.813370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:00.813370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.813370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:00.813370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:00.813370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.813370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:00.813370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:00.813370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:00.813370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.813726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:00.813726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:00.813726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:00.813726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:00.813726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:00.813726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:00.813726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:00.813726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:00.813726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:00.813726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:00.813726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:00.813726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:00.813726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:00.814112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.814112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:00.814412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.814412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:00.814412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:00.814412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:00.814412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:00.814412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:00.814412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.814412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:00.814412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:00.814412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.814412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:00.814412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:00.814412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.814412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:00.814412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:00.814412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:00.814857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:00.814857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:00.814857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.814857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:00.814857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:00.814857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:00.814857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:00.814857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:00.814857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:00.814857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:00.814857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:00.814857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:00.814857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:00.814857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:00.814857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:00.814857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:00.814857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:00.814857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:00.814857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:00.815413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:00.815413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:00.815413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:03.697696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:03.698095 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:03.698095 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:03.698752 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:03.741904 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:03.741988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:03.742402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:03.742402 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:03.742402 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:03.742402 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:03.742402 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:03.742402 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:03.742402 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:03.742402 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:03.743142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:03.743142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:03.743142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:03.743142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:03.743142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:03.743142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:03.743142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:03.743142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:03.743142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:03.743142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:03.743142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:03.743142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:03.743401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:03.743401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:03.743401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:03.743401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:03.743401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:03.743401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:03.743401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:03.743401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:03.743401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:03.743401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:03.743401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:03.743401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:03.743401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:03.743633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:03.743633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:03.744105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:03.744105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:03.744105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:03.744105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:03.744105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:03.744105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:03.744105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:03.744105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:03.744105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:03.744105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:03.744105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:03.744105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:03.744105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:03.744105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:03.744105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:03.744105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:03.744440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:03.744440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:03.744440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:03.744440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:03.744440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:03.744440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:03.744440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:03.744440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:03.744440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:03.744440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:03.744440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:03.744440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:03.744440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:03.744440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:03.744440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:03.744440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:03.744440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:03.744440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:03.744440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:03.744761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:03.744761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:03.744761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:05.765908 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:05.766554 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:05.766554 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:05.767654 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:05.767911 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-acd0c4a8-21a2-4e59-8c73-9c6fb220d996, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:05.768080 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:05.768225 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:05.768359 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:05.768463 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:05.768567 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e18214a6-42af-4a82-ae91-b005b582cf79, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:05.768665 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:05.768768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.769365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:05.769365 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:05.769365 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:05.769365 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:05.769365 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:05.769365 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:05.769365 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.769365 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:05.770703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.770703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:05.770703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:05.770703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:05.770703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.770703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:05.770703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:05.770703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.770703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:05.770703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:05.770703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:05.770703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.771111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:05.771111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:05.771111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:05.771111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:05.771111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:05.771111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:05.771111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:05.771111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:05.771111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:05.771111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:05.771111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:05.771111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:05.771111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:05.771496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.771496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:05.771496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.771496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:05.771496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:05.771496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:05.771496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:05.771496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:05.771496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.771496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:05.771496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:05.771496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.771496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:05.771496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:05.771496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.771806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:05.771806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:05.771806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.771806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:05.771806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:05.771806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.771806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:05.771806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:05.771806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:05.771806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.771806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:05.771806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:05.771806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:05.771806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:05.771806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:05.771806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:05.771806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:05.771806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:05.771806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:05.772164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:05.772164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:05.772164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:05.772164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:05.772164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.772164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:05.791735 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:05.792259 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:05.792370 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:05.792467 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:32:05.792788 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:05.792951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.794317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:05.794317 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:05.794317 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:05.794317 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:05.794317 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:05.794317 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:05.794317 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.794317 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:05.795243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.795243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:05.795243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:05.795243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:05.795243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.795243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:05.795243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:05.795243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:05.795243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.795243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:05.795243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:05.795243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:05.795243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:05.795243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:05.795243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:05.795567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:05.795567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:05.795567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:05.795567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:05.795567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:05.795567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:05.795567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:05.795567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.795567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:05.796217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.796217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:05.796217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:05.796217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:05.796217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.796217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:05.796217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:05.796217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:32:05.796217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:05.796217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:05.796217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.796217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:05.796486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:05.796486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:05.796486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.796486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:05.796486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:05.796486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:05.796486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:05.796486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:05.796486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:05.796486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:05.796486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:05.796486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:05.796486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:05.796486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:05.796718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:05.796718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:05.796718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.796718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:05.797403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.797403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:05.797403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:05.797403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:05.797403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:05.797403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:05.797403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.797403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:05.797403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:05.797403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.797403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:05.797403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:05.797403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.797403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:05.797403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:05.797403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.797723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:05.797723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:05.797723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:32:05.797723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:05.797723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:05.797723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.797723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:05.797723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:05.797723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:05.797723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.797723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:05.797723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:05.797723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:05.797723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:05.797723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:05.797723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:05.797723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:05.797723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:05.798057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:05.798057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:05.798057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:05.798057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:05.798057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:05.798057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.798057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:05.806732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:05.806945 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:05.807058 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:05.807173 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:05.807540 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:05.807811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.808485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:05.808485 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:05.808485 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:05.808485 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:05.808485 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:05.808485 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:05.808485 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.808485 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:05.809183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.809183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:05.809183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:05.809183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:05.809183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.809183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:05.809183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:05.809183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:05.809183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.809183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:05.809183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:05.809183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:05.809183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:05.809183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:05.809183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:05.809542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:05.809542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:05.809542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:05.809542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:05.809542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:05.809542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:05.809542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:05.809542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.809542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:05.810003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.810003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:05.810003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:05.810003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:05.810003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.810003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:05.810003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:05.810003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:05.810003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:05.810003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.810003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:05.810234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:05.810234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:05.810234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.810234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:05.810234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:05.810234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:05.810234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:05.810234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:05.810234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:05.810234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:05.810234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:05.810234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:05.810234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:05.810234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:05.810513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:05.810513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:05.810513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.810513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:05.811314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.811314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:05.811314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:05.811314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:05.811314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:05.811314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:05.811314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.811314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:05.811314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:05.811314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.811314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:05.811314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:05.811314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.811314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:05.811314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:05.811314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.811314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:05.811917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:05.811917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:05.811917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:05.811917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.811917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:05.811917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:05.811917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:05.811917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.811917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:05.811917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:05.811917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:05.811917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:05.811917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:05.811917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:05.811917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:05.811917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:05.811917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:05.811917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:05.811917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:05.812535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:05.812535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:05.812535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.812535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:05.812535 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:05.812535 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:05.812535 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:05.812535 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:05.812814 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:05.812814 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:05.812814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.813364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:05.813364 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:05.813364 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:05.813364 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:05.813364 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:05.813364 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:05.813364 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.813364 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:05.814106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.814106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:05.814106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:05.814106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:05.814106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.814106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:05.814106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:05.814106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.814106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:05.814106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:05.814106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:05.814106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.814544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:05.814544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:05.814544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:05.814544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:05.814544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:05.814544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:05.814544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:05.814544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:05.814544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:05.814544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:05.814544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:05.814544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:05.814544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:05.814984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.814984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:05.815068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.815068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:05.815068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:05.815068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:05.815068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:05.815068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:05.815068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.815068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:05.815068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:05.815068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.815068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:05.815068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:05.815068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.815068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:05.815068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:05.815068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:05.815667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:05.815667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:05.815667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.815667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:05.815667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:05.815667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:05.815667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:05.815667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:05.815667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:05.815667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:05.815667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:05.815667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:05.815667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:05.815667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:05.815667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:05.815667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:05.815667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:05.815667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:05.815667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:05.816271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:05.816271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:05.816271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:08.701682 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:08.701682 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:08.701682 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:08.701682 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:08.755032 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:08.755345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:08.756054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:08.756054 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:08.756054 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:08.756054 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:08.756054 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:08.756054 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:08.756054 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:08.756054 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:08.757146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:08.757146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:08.757146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:08.757146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:08.757146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:08.757146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:08.757146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:08.757146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:08.757146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:08.757146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:08.757146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:08.757146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:08.757538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:08.757538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:08.757538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:08.757538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:08.757538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:08.757538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:08.757538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:08.757538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:08.757538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:08.757538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:08.757538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:08.757538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:08.757538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:08.757890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:08.757890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:08.758299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:08.758299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:08.758299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:08.758299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:08.758299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:08.758299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:08.758299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:08.758299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:08.758299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:08.758299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:08.758299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:08.758299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:08.758299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:08.758299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:08.758299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:08.758299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:08.758769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:08.758769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:08.758769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:08.758769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:08.758769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:08.758769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:08.758769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:08.758769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:08.758769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:08.758769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:08.758769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:08.758769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:08.758769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:08.758769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:08.758769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:08.758769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:08.758769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:08.758769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:08.758769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:08.759351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:08.759351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:08.759351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:10.766556 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:10.767708 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:10.767708 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:10.768718 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:10.769035 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:10.769264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:10.770880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:10.771075 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:10.771246 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:10.771435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.773637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:10.773637 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:10.773637 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:10.773637 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:10.773637 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:10.773637 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:10.773637 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.773637 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:10.773637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.773637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:10.773637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:10.773637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:10.773637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.773637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:10.773637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:10.773955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.773955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:10.773955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:10.773955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:10.773955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.773955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:10.773955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:10.773955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:10.773955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:10.773955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:10.773955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:10.773955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:10.773955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:10.773955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:10.774224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:10.774224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:10.774224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:10.774224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:10.774224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.774224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:10.774224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.774224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:10.774224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:10.774224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:10.774224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:10.774224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:10.774224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.774224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:10.774500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:10.774860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:10.774860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:10.774860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:10.774860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:10.774860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:10.774860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:10.774860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:10.774860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:10.774860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.774860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:10.792348 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:10.792849 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:10.793083 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:10.793667 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:32:10.794692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:10.795687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.798341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:10.798341 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:10.798341 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:10.798341 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:10.798341 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:10.798341 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:10.798341 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.798341 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:10.798341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.798341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:10.798341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:10.798341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:10.798341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.798341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:10.798341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:10.798341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:10.798341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.798341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:10.798945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:10.798945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:10.798945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:10.798945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:10.798945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:10.798945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:10.798945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:10.798945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:10.798945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:10.798945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:10.798945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:10.798945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:10.798945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.798945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:10.798945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.799281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:10.799281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:10.799281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:10.799281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.799281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:10.799281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:10.799281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:32:10.799281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:10.799281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:10.799281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.799281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:10.799281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:10.799281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:10.799497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.799497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:10.799497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:10.799497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:10.799497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:10.799497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:10.799497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:10.799497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:10.799497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:10.799497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:10.799497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:10.799497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:10.799497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:10.799497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:10.799785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.799785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:10.799785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.799785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:10.799785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:10.799785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:10.799785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:10.799785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:10.799785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.799785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:10.799785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:10.799785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.799785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:10.799785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:10.799785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.800122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:10.800122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:10.800122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.800122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:10.800122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:10.800122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:32:10.800122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:10.800122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:10.800122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.800122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:10.800122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:10.800122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:10.800122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.800122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:10.800122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:10.800122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:10.800122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:10.800122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:10.800122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:10.800452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:10.800452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:10.800452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:10.800452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:10.800452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:10.800452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:10.800452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:10.800452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.800452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:10.807130 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:10.807774 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:10.807945 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:10.808206 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:10.808637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:10.808792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.813202 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:10.816012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:10.816012 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:10.816012 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:10.816012 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:10.816012 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:10.816012 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:10.816012 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.816012 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:10.816012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.816012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:10.816012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:10.816012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:10.816012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.816012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:10.816012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:10.816012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:10.816012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.816012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:10.816392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:10.816392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:10.816392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:10.816392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:10.816392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:10.816392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:10.816392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:10.816392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:10.816392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:10.816392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:10.816392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:10.816392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:10.816392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.816392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:10.816392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.816655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:10.816655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:10.816655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:10.816655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.816655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:10.816655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:10.816655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:10.816655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:10.816655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.816655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:10.816655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:10.816655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:10.816655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.816874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:10.816874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:10.816874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:10.816874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:10.816874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:10.816874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:10.816874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:10.816874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:10.816874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:10.816874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:10.816874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:10.816874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:10.816874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:10.817129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.817129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:10.817129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.817129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:10.817129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:10.817129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:10.817129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:10.817129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:10.817129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.817129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:10.817129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:10.817129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.817129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:10.817129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:10.817129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.817418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:10.817418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:10.817418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.817418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:10.817418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:10.817418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:10.817418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:10.817418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.817418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:10.817418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:10.817418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:10.817418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.817418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:10.817418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:10.817418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:10.817418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:10.817418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:10.817418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:10.817418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:10.817720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:10.817720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:10.817720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:10.817720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:10.817720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:10.817720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:10.817720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.817720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:10.817720 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:10.817720 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:10.817720 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:10.817915 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:10.817915 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:10.817915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.818837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:10.818837 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:10.818837 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:10.818837 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:10.818837 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:10.818837 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:10.818837 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.818837 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:10.819734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.819734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:10.819734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:10.819734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:10.819734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.819734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:10.819734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:10.819734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.819734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:10.819734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:10.819734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:10.819734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.820312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:10.820312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:10.820312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:10.820312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:10.820312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:10.820312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:10.820312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:10.820312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:10.820312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:10.820312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:10.820312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:10.820312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:10.820312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:10.820565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.820565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:10.821913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.821913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:10.821913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:10.821913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:10.821913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:10.821913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:10.821913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.821913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:10.821913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:10.821913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.821913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:10.821913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:10.821913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.821913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:10.821913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:10.821913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:10.822591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:10.822591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:10.822591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.822591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:10.822591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:10.822591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:10.822591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:10.822591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:10.822591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:10.822591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:10.822591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:10.822591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:10.822591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:10.822591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:10.822591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:10.822591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:10.822591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:10.822591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:10.822591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:10.822933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:10.822933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:10.822933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:13.698515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:13.701213 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:13.701213 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:13.701213 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:13.735966 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:13.736305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:13.736762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:13.736762 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:13.736762 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:13.736762 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:13.736762 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:13.736762 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:13.736762 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:13.736762 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:13.737379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:13.737379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:13.737379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:13.737379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:13.737379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:13.737379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:13.737379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:13.737379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:13.737379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:13.737379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:13.737379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:13.737379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:13.737608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:13.737608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:13.737608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:13.737608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:13.737608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:13.737608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:13.737608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:13.737608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:13.737608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:13.737608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:13.737608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:13.737608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:13.737608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:13.737804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:13.737804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:13.738278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:13.738278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:13.738278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:13.738278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:13.738278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:13.738278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:13.738278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:13.738278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:13.738278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:13.738278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:13.738278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:13.738278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:13.738278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:13.738278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:13.738278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:13.738278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:13.738594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:13.738594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:13.738594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:13.738594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:13.738594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:13.738594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:13.738594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:13.738594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:13.738594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:13.738594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:13.738594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:13.738594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:13.738594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:13.738594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:13.738594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:13.738594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:13.738594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:13.738594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:13.738594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:13.738944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:13.738944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:13.738944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:15.767263 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:15.767825 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:15.767825 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:15.768631 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:15.768840 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:15.768987 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:15.769098 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:15.769211 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:15.769322 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:15.769419 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:15.769519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.769965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:15.769965 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:15.769965 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:15.769965 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:15.769965 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:15.769965 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:15.769965 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.769965 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:15.770713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.770713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:15.770713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:15.770713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:15.770713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.770713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:15.770713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:15.770713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.770713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:15.770713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:15.770713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:15.770713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.771177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:15.771177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:15.771177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:15.771177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:15.771177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:15.771177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:15.771177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:15.771177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:15.771177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:15.771177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:15.771177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:15.771177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:15.771177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:15.771640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.771640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:15.771640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.771640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:15.771640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:15.771640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:15.771640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:15.771640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:15.771640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.771640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:15.771640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:15.771640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.771640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:15.771640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:15.771640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.772163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:15.772163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:15.772163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.772163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:15.772163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:15.772163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.772163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:15.772163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:15.772163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:15.772163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.772163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:15.772163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:15.772163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:15.772163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:15.772163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:15.772163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:15.772163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:15.772163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:15.772163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:15.772783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:15.772783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:15.772783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:15.772783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:15.772783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.772783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:15.792852 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:15.793202 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:15.793311 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:15.793420 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:32:15.796665 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:15.796866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.797344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:15.797344 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:15.797344 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:15.797344 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:15.797344 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:15.797344 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:15.797344 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.797344 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:15.797881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.797881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:15.797881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:15.797881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:15.797881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.797881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:15.797881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:15.797881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:15.797881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.797881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:15.797881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:15.797881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:15.797881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:15.797881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:15.797881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:15.798194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:15.798194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:15.798194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:15.798194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:15.798194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:15.798194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:15.798194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:15.798194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.798194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:15.798492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.798492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:15.798492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:15.798492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:15.798492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.798492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:15.798492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:15.798492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:32:15.798492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:15.798492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:15.798492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.798492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:15.798717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:15.798717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:15.798717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.798717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:15.798717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:15.798717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:15.798717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:15.798717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:15.798717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:15.798717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:15.798717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:15.798717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:15.798717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:15.798717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:15.798951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:15.798951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:15.798951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.798951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:15.799210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.799210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:15.799210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:15.799210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:15.799210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:15.799210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:15.799210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.799210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:15.799210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:15.799210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.799210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:15.799210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:15.799210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.799210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:15.799210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:15.799210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.799506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:15.799506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:15.799506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:32:15.799506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:15.799506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:15.799506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.799506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:15.799506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:15.799506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:15.799506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.799506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:15.799506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:15.799506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:15.799506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:15.799506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:15.799506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:15.799506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:15.799506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:15.799798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:15.799798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:15.799798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:15.799798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:15.799798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:15.799798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.799798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:15.807776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:15.808010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:15.808098 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:15.808207 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:15.809169 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:15.809410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.810449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:15.810449 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:15.810449 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:15.810449 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:15.810449 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:15.810449 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:15.810449 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.810449 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:15.810997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.810997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:15.810997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:15.810997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:15.810997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.810997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:15.810997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:15.810997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:15.810997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.810997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:15.810997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:15.810997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:15.810997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:15.810997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:15.810997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:15.811268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:15.811268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:15.811268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:15.811268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:15.811268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:15.811268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:15.811268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:15.811268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.811268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:15.811798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.811798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:15.811798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:15.811798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:15.811798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.811798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:15.811798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:15.811798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:15.811798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:15.811798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.811798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:15.812066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:15.812066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:15.812066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.812066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:15.812066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:15.812066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:15.812066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:15.812066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:15.812066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:15.812066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:15.812066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:15.812066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:15.812066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:15.812066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:15.812324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:15.812324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:15.812324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.812324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:15.812573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.812573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:15.812573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:15.812573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:15.812573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:15.812573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:15.812573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.812573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:15.812573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:15.812573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.812573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:15.812573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:15.812573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.812573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:15.812573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:15.812573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.812573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:15.812931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:15.812931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:15.812931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:15.812931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.812931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:15.812931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:15.812931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:15.812931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.812931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:15.812931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:15.812931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:15.812931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:15.812931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:15.812931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:15.812931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:15.812931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:15.812931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:15.812931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:15.812931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:15.813285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:15.813285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:15.813285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.813285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:15.815908 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:15.816178 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:15.816301 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:15.816442 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:15.816909 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:15.817226 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:15.817428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.817996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:15.817996 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:15.817996 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:15.817996 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:15.817996 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:15.817996 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:15.817996 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.817996 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:15.818880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.818880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:15.818880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:15.818880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:15.818880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.818880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:15.818880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:15.818880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.818880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:15.818880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:15.818880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:15.818880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.819140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:15.819140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:15.819140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:15.819140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:15.819140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:15.819140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:15.819140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:15.819140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:15.819140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:15.819140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:15.819140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:15.819140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:15.819140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:15.819375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.819375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:15.819577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.819577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:15.819577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:15.819577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:15.819577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:15.819577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:15.819577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.819577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:15.819577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:15.819577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.819577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:15.819577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:15.819577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.819577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:15.819577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:15.819577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:15.819843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:15.819843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:15.819843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.819843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:15.819843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:15.819843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:15.819843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:15.819843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:15.819843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:15.819843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:15.819843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:15.819843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:15.819843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:15.819843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:15.819843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:15.819843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:15.819843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:15.819843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:15.819843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:15.820175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:15.820175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:15.820175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:18.700542 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:18.701571 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:18.701571 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:18.702485 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:18.758670 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:18.758670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:18.759004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:18.759004 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:18.759004 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:18.759004 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:18.759004 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:18.759004 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:18.759004 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:18.759004 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:18.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.057 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:18.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:18.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:18.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:18.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:18.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:18.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:18.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:18.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:18.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:18.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:18.760199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:18.760599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:18.760599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:18.760599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:18.760599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:18.760599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:18.760599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:18.760599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:18.760599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:18.760599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:18.760599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:18.760599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:18.760599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:18.760599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:18.760963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:18.760963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:18.761814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:18.761814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:18.761814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:18.761814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:18.761814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:18.761814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:18.761814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:18.761814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:18.761814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:18.761814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:18.761814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:18.761814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:18.761814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:18.761814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:18.761814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:18.761814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:18.763110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:18.763110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:18.763110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:18.763110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:18.763110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:18.763110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:18.763110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:18.763110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:18.763110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:18.763110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:18.763110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:18.763110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:18.763110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:18.763110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:18.763110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:18.763110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:18.763110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:18.763110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:18.763110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:18.763670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:18.763670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:18.763670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:20.769345 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:20.770757 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:20.770955 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:20.773538 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:20.773838 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:20.774107 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:20.774429 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:20.774711 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:20.774922 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:20.775139 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:20.776518 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:20.776748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.778432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:20.778432 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:20.778432 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:20.778432 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:20.778432 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:20.778432 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:20.778432 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.778432 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:20.779658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.779658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:20.779658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:20.779658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:20.779658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.779658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:20.779658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:20.779658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.779658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:20.779658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:20.779658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:20.779658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.780055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:20.780055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:20.780055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:20.780055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:20.780055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:20.780055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:20.780055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:20.780055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:20.780055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:20.780055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:20.780055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:20.780055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:20.780055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:20.780409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.780409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:20.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:20.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:20.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:20.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:20.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:20.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:20.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:20.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:20.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:20.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:20.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:20.784485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.784986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:20.784986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:20.784986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.784986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:20.784986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:20.784986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:20.784986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.784986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:20.784986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:20.784986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:20.784986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:20.784986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:20.784986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:20.784986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:20.784986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:20.784986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:20.784986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:20.784986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:20.784986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:20.786450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:20.786450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.786450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:20.793521 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:20.793934 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:20.794554 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:20.794554 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:32:20.794670 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:20.794790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.796604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:20.796604 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:20.796604 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:20.796604 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:20.796604 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:20.796604 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:20.796604 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.796604 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:20.797333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.797333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:20.797333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:20.797333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:20.797333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.797333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:20.797333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:20.797333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:20.797333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.797333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:20.797333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:20.797333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:20.797333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:20.797333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:20.797333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:20.797750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:20.797750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:20.797750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:20.797750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:20.797750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:20.797750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:20.797750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:20.797750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.797750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:20.798323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.798323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:20.798323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:20.798323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:20.798323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.798323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:20.798323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:20.798323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:32:20.798323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:20.798323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:20.798323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.798323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:20.798710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:20.798710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:20.798710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.798710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:20.798710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:20.798710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:20.798710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:20.798710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:20.798710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:20.798710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:20.798710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:20.798710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:20.798710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:20.798710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:20.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:20.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:20.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:20.808067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.808067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:20.808067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:20.808067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:20.808067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:20.808067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:20.808067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.808067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:20.808067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:20.808067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.808067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:20.808067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:20.808067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.808067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:20.808067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:20.808067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.808902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:20.808902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:20.808902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:32:20.808902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:20.808902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:20.808902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.808902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:20.808902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:20.808902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:20.808902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.808902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:20.808902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:20.808902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:20.808902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:20.808902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:20.808902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:20.808902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:20.808902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:20.809690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:20.809690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:20.809690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:20.809690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:20.809690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:20.809690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.809690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:20.809690 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:20.809879 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:20.810487 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:20.810487 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:20.814433 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:20.814513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.816727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:20.816727 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:20.816727 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:20.816727 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:20.816727 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:20.816727 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:20.816727 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.816727 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:20.817038 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:20.818040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.818040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:20.818040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:20.818040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:20.818040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.818040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:20.818040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:20.818040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:20.818040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.818040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:20.818040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:20.818040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:20.818040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:20.818040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:20.818040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:20.818420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:20.818420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:20.818420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:20.818420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:20.818420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:20.818420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:20.818420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:20.818420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.818420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:20.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:20.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:20.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:20.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:20.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:20.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:20.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:20.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:20.820383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:20.820383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:20.820383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.820383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:20.820383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:20.820383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:20.820383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:20.820383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:20.820383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:20.820383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:20.820383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:20.820383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:20.820383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:20.820383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:20.820717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:20.820717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:20.820717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.820717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:20.821363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.821363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:20.821363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:20.821363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:20.821363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:20.821363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:20.821363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.821363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:20.821363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:20.821363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.821363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:20.821363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:20.821363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.821363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:20.821363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:20.821363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.821363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:20.821676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:20.821676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:20.821676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:20.821676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.821676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:20.821676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:20.821676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:20.821676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.821676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:20.821676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:20.821676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:20.821676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:20.821676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:20.821676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:20.821676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:20.821676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:20.821676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:20.821676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:20.821676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:20.822002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:20.822002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:20.822002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.822002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:20.822002 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:20.822002 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:20.822002 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:20.823026 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:20.824124 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:20.824124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.824931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:20.824931 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:20.824931 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:20.824931 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:20.824931 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:20.824931 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:20.824931 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.824931 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:20.826087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.826087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:20.826087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:20.826087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:20.826087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.826087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:20.826087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:20.826087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.826087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:20.826087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:20.826087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:20.826087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.826349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:20.826349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:20.826349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:20.826349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:20.826349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:20.826349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:20.826349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:20.826349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:20.826349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:20.826349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:20.826349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:20.826349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:20.826349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:20.826560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.826560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:20.828296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.828296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:20.828296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:20.828296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:20.828296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:20.828296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:20.828296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.828296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:20.828296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:20.828296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.828296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:20.828296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:20.828296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.828296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:20.828296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:20.828296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:20.828620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:20.828620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:20.828620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.828620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:20.828620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:20.828620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:20.828620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:20.828620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:20.828620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:20.828620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:20.828620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:20.828620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:20.828620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:20.828620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:20.828620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:20.828620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:20.828620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:20.828620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:20.828620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:20.828994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:20.828994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:20.828994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:23.701115 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:23.703605 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:23.703605 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:23.703605 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:23.747089 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:23.747328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:23.748046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:23.748046 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:23.748046 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:23.748046 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:23.748046 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:23.748046 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:23.748046 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:23.748046 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:23.750707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:23.750707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:23.750707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:23.750707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:23.750707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:23.750707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:23.750707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:23.750707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:23.750707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:23.750707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:23.750707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:23.750707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:23.750939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:23.750939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:23.750939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:23.750939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:23.750939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:23.750939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:23.750939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:23.750939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:23.750939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:23.750939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:23.750939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:23.750939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:23.750939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:23.751180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:23.751180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:23.753813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:23.753813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:23.753813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:23.753813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:23.753813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:23.753813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:23.753813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:23.753813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:23.753813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:23.753813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:23.753813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:23.753813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:23.753813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:23.753813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:23.753813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:23.753813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:23.754338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:23.754338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:23.754338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:23.754338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:23.754338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:23.754338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:23.754338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:23.754338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:23.754338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:23.754338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:23.754338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:23.754338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:23.754338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:23.754338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:23.754338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:23.754338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:23.754338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:23.754338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:23.754338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:23.754915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:23.754915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:23.754915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:25.770807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:25.772270 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:25.772270 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:25.772637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:25.772834 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:25.772955 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:25.773101 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:25.773205 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:25.773336 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:25.773433 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:25.773531 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:25.773633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.775559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:25.775559 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:25.775559 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:25.775559 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:25.775559 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:25.775559 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:25.775559 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.775559 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:25.775559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.775559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:25.775559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:25.775559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:25.775559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.775559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:25.775559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:25.776066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.776066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:25.776066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:25.776066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:25.776066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.776066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:25.776066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:25.776066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:25.776066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:25.776066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:25.776066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:25.776066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:25.776066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:25.776066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:25.776495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:25.776495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:25.776495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:25.776495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:25.776495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.776495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:25.776495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.776495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:25.776495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:25.776495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:25.776495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:25.776495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:25.776495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.776495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:25.776913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:25.777583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:25.777583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:25.777583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:25.777583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:25.777583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:25.777583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:25.777583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:25.777583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:25.777583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.777583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:25.795792 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:25.797066 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:25.797066 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:25.797066 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:32:25.797381 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:25.797561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.798025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:25.798025 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:25.798025 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:25.798025 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:25.798025 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:25.798025 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:25.798025 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.798025 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:25.800979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.800979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:25.800979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:25.800979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:25.800979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.800979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:25.800979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:25.800979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:25.800979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.800979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:25.800979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:25.800979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:25.800979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:25.800979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:25.800979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:25.801341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:25.801341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:25.801341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:25.801341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:25.801341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:25.801341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:25.801341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:25.801341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.801341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:25.801341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.801341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:25.801341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:25.801341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:25.801341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.801601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:25.801601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:25.801601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:32:25.801601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:25.801601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:25.801601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.801601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:25.801601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:25.801601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:25.801601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.801601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:25.801601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:25.801601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:25.801797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:25.801797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:25.801797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:25.801797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:25.801797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:25.801797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:25.801797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:25.801797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:25.801797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:25.801797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:25.801797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.801797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:25.802022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.802022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:25.802022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:25.802022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:25.802022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:25.802022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:25.802022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.802022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:25.802022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:25.802022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.802022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:25.802022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:25.802022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.802022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:25.802022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:25.802022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.802310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:25.802310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:25.802310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:32:25.802310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:25.802310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:25.802310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.802310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:25.802310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:25.802310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:25.802310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.802310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:25.802310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:25.802310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:25.802310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:25.802310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:25.802310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:25.802310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:25.802310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:25.802613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:25.802613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:25.802613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:25.802613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:25.802613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:25.802613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.802613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:25.810736 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:25.812578 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:25.813818 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:25.813818 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:25.814221 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:25.814315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.814706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:25.814706 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:25.814706 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:25.814706 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:25.814706 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:25.814706 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:25.814706 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.814706 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:25.815941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.815941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:25.815941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:25.815941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:25.815941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.815941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:25.815941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:25.815941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:25.815941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.815941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:25.815941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:25.815941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:25.815941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:25.815941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:25.815941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:25.816286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:25.816286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:25.816286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:25.816286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:25.816286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:25.816286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:25.816286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:25.816286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.816286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:25.819332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.819332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:25.819332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:25.819332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:25.819332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.819332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:25.819332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:25.819332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:25.819332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:25.819332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.819332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:25.819636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:25.819636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:25.819636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.819636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:25.819636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:25.819636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:25.819636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:25.819636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:25.819636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:25.819636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:25.819636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:25.819636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:25.819636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:25.819636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:25.819916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:25.819916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:25.819916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.819916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:25.820482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.820482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:25.820482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:25.820482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:25.820482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:25.820482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:25.820482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.820482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:25.820482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:25.820482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.820482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:25.820482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:25.820482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.820482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:25.820482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:25.820482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.820482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:25.820802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:25.820802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:25.820802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:25.820802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.820802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:25.820802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:25.820802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:25.820802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.820802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:25.820802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:25.820802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:25.820802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:25.820802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:25.820802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:25.820802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:25.820802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:25.820802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:25.820802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:25.820802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:25.821563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:25.821563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:25.821563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.821563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:25.821811 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:25.822290 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:25.822290 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:25.823244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:25.823562 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:25.823770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:25.823968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.824547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:25.824547 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:25.824547 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:25.824547 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:25.824547 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:25.824547 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:25.824547 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.824547 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:25.825205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.825205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:25.825205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:25.825205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:25.825205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.825205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:25.825205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:25.825205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.825205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:25.825205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:25.825205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:25.825205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.825456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:25.825456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:25.825456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:25.825456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:25.825456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:25.825456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:25.825456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:25.825456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:25.825456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:25.825456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:25.825456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:25.825456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:25.825456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:25.825643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.825643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:25.826004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.826004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:25.826004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:25.826004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:25.826004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:25.826004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:25.826004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.826004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:25.826004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:25.826004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.826004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:25.826004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:25.826004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.826004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:25.826004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:25.826004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:25.826284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:25.826284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:25.826284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.826284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:25.826284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:25.826284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:25.826284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:25.826284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:25.826284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:25.826284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:25.826284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:25.826284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:25.826284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:25.826284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:25.826284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:25.826284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:25.826284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:25.826284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:25.826284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:25.826566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:25.826566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:25.826566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:28.702555 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:28.703431 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:28.703547 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:28.707096 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:28.752414 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:28.754714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:28.757572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:28.757572 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:28.757572 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:28.757572 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:28.757572 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:28.757572 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:28.757572 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:28.757572 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:28.758345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.054 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:28.758345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:28.758345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:28.758345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:28.758345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:28.758345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:28.758345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:28.758345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:28.758345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:28.758345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:28.758345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:28.758345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:28.759002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:28.759002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:28.759002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:28.759002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:28.759002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:28.759002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:28.759002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:28.759002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:28.759002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:28.759002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:28.759002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:28.759002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:28.759002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:28.759652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:28.759652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:28.759652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:28.759652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:28.759652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:28.759652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:28.759652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:28.759652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:28.759652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:28.759652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:28.759652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:28.759652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:28.759652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:28.759652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:28.759652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:28.760170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:28.760170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:28.760170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:28.760170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:28.760170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:28.760170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:28.760170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:28.760170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:28.760170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:28.760170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:28.760170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:28.760170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:28.760170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:28.760170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:28.760170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:28.760170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:28.760170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:28.760170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:28.760170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:28.761148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:28.761148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:28.761148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:28.761148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:28.761148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:28.761148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:30.771727 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:30.772290 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:30.772537 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:30.774556 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:30.775145 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:30.775145 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:30.775145 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:30.775145 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:30.775145 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:30.775387 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:30.775439 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:30.775497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.776072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:30.776072 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:30.776072 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:30.776072 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:30.776072 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:30.776072 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:30.776072 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.776072 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:30.778374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.778374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:30.778374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:30.778374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:30.778374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.778374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:30.778374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:30.778374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.778374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:30.778374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:30.778374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:30.778374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.778629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:30.778629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:30.778629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:30.778629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:30.778629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:30.778629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:30.778629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:30.778629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:30.778629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:30.778629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:30.778629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:30.778629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:30.778629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:30.778833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.778833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:30.778833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.778833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:30.778833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:30.778833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:30.778833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:30.778833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:30.778833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.778833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:30.778833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:30.778833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.778833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:30.778833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:30.778833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.779090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:30.779090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:30.779090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.779090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:30.779090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:30.779090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.779090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:30.779090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:30.779090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:30.779090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.779090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:30.779090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:30.779090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:30.779090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:30.779090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:30.779090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:30.779090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:30.779090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:30.779090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:30.779407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:30.779407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:30.779407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:30.779407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:30.779407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.779407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:30.796941 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:30.797362 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:30.797554 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:30.797822 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:32:30.798669 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:30.798669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.798669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:30.798669 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:30.798669 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:30.798669 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:30.798669 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:30.798669 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:30.798669 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.798669 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:30.799284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.799284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:30.799284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:30.799284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:30.799284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.799284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:30.799284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:30.799284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:30.799284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.799284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:30.799284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:30.799284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:30.799284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:30.799284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:30.799284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:30.799527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:30.799527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:30.799527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:30.799527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:30.799527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:30.799527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:30.799527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:30.799527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.799527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:30.800107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.800107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:30.800107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:30.800107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:30.800107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.800107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:30.800107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:30.800107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:32:30.800107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:30.800107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:30.800107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.800107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:30.800377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:30.800377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:30.800377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.800377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:30.800377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:30.800377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:30.800377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:30.800377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:30.800377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:30.800377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:30.800377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:30.800377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:30.800377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:30.800377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:30.800589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:30.800589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:30.800589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.800589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:30.800589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.800589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:30.800589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:30.800589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:30.800589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:30.800589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:30.800589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.800589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:30.800589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:30.800589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.800809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:30.800809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:30.800809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.800809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:30.800809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:30.800809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.800809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:30.800809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:30.800809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:32:30.800809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:30.800809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:30.800809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.800809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:30.800809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:30.800809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:30.800809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.800809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:30.800809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:30.800809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:30.801141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:30.801141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:30.801141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:30.801141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:30.801141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:30.801141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:30.801141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:30.801141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:30.801141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:30.801141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:30.801141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.801141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:30.814385 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:30.814385 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:30.814385 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:30.814385 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:30.815524 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:30.817270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.820622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:30.820622 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:30.820622 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:30.820622 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:30.820622 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:30.820622 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:30.820622 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.820622 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:30.826848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.826848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:30.826848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:30.826848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:30.826848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.826848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:30.826848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:30.826848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:30.826848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.826848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:30.826848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:30.826848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:30.826848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:30.826848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:30.826848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:30.831030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:30.831030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:30.831030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:30.831030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:30.831030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:30.831030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:30.831030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:30.831030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.831030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:30.831030 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:30.831030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.013 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.831030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:30.831423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:30.831423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:30.831423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.831423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:30.831423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:30.831423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:30.831423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:30.831423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.831423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:30.831423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:30.831423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:30.831423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.831423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:30.831757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:30.831757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:30.831757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:30.831757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:30.831757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:30.831757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:30.831757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:30.831757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:30.831757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:30.831757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:30.831757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:30.831757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:30.831757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.832025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:30.832025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.832025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:30.832025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:30.832025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:30.832025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:30.832025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:30.832025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.832025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:30.832025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:30.832025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.832025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:30.832025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:30.832025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.832025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:30.832025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:30.832025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.832355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:30.832355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:30.832355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:30.832355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:30.832355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.832355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:30.832355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:30.832355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:30.832355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.832355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:30.832355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:30.832355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:30.832355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:30.832355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:30.832355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:30.832355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:30.832355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:30.832355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:30.832355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:30.832678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:30.832678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:30.832678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:30.832678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.832678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:30.832678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:30.832678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:30.832678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:30.832678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:30.832867 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:30.833999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.834611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:30.834611 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:30.834611 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:30.834611 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:30.834611 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:30.834611 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:30.834611 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.834611 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:30.836127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.836127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:30.836127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:30.836127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:30.836127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.836127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:30.836127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:30.836127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.836127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:30.836127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:30.836127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:30.836127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.837213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:30.837213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:30.837213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:30.837213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:30.837213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:30.837213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:30.837213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:30.837213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:30.837213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:30.837213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:30.837213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:30.837213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:30.837213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:30.837874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.837874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:30.841854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.841854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:30.841854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:30.841854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:30.841854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:30.841854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:30.841854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.841854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:30.841854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:30.841854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.841854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:30.841854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:30.841854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.841854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:30.841854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:30.841854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:30.842412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:30.842412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:30.842412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.842412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:30.842412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:30.842412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:30.842412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:30.842412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:30.842412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:30.842412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:30.842412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:30.842412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:30.842412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:30.842412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:30.842412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:30.842412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:30.842412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:30.842412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:30.842412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:30.842897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:30.842897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:30.842897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:33.704531 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:33.706375 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:33.706375 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:33.706375 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:33.738819 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:33.739037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:33.741089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:33.741089 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:33.741089 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:33.741089 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:33.741089 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:33.741089 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:33.741089 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:33.741089 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:33.741089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:33.741089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:33.741089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:33.741089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:33.741089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:33.741089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:33.741089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:33.741695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:33.741695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:33.741695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:33.741695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:33.741695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:33.741695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:33.741695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:33.741695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:33.741695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:33.741695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:33.741695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:33.741695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:33.741695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:33.741695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:33.742107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:33.742107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:33.742107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:33.742107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:33.742107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:33.742107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:33.742107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:33.742107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:33.742107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:33.742107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:33.742107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:33.742107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:33.742107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:33.742107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:33.742399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:33.743046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:33.743046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:33.743046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:33.743046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:33.743046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:33.743046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:33.743046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:33.743046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:33.743046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:35.772174 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:35.773769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:35.773769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:35.774028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:35.774224 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:35.774388 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:35.774543 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:35.774673 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:35.774810 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:35.774938 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:35.775083 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:35.775218 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-27694673-a08c-41e6-8636-2613670955d3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:35.775366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.777396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:35.777396 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:35.777396 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:35.777396 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:35.777396 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:35.777396 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:35.777396 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.777396 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:35.777396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.777396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:35.777396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:35.777396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:35.777396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.777396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:35.777396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:35.777760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.777760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:35.777760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:35.777760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:35.777760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.777760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:35.777760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:35.777760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:35.777760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:35.777760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:35.777760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:35.777760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:35.777760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:35.777760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:35.778078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:35.778078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:35.778078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:35.778078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:35.778078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.778078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:35.778078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.778078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:35.778078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:35.778078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:35.778078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:35.778078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:35.778078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.778078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:35.778365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:35.778763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:35.778763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:35.778763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:35.778763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:35.778763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:35.778763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:35.778763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:35.778763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:35.778763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.778763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:35.797431 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:35.798222 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:35.798222 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:35.798222 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:32:35.798550 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:35.798745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.801400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:35.801400 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:35.801400 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:35.801400 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:35.801400 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:35.801400 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:35.801400 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.801400 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:35.801400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.801400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:35.801400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:35.801400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:35.801400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.801400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:35.801400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:35.801400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:35.801400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.801400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:35.801803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:35.801803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:35.801803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:35.801803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:35.801803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:35.801803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:35.801803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:35.801803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:35.801803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:35.801803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:35.801803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:35.801803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:35.801803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.801803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:35.801803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.802159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:35.802159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:35.802159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:35.802159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.802159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:35.802159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:35.802159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:32:35.802159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:35.802159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:35.802159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.802159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:35.802159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:35.802159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:35.802543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.802543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:35.802543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:35.802543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:35.802543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:35.802543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:35.802543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:35.802543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:35.802543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:35.802543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:35.802543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:35.802543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:35.802543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:35.802543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:35.802809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.802809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:35.802809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.802809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:35.802809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:35.802809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:35.802809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:35.802809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:35.802809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.802809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:35.802809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:35.802809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.802809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:35.802809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:35.802809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.803105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:35.803105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:35.803105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.803105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:35.803105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:35.803105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:32:35.803105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:35.803105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:35.803105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.803105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:35.803105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:35.803105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:35.803105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.803105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:35.803105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:35.803105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:35.803105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:35.803105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:35.803105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:35.803495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:35.803495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:35.803495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:35.803495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:35.803495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:35.803495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:35.803495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:35.803495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.803495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:35.812998 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:35.813813 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:35.813813 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:35.813813 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:35.814083 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:35.814318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.816843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:35.816843 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:35.816843 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:35.816843 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:35.816843 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:35.816843 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:35.816843 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.816843 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:35.816843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.816843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:35.816843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:35.816843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:35.816843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.816843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:35.816843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:35.816843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:35.816843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.816843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:35.817376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:35.817376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:35.817376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:35.817376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:35.817376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:35.817376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:35.817376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:35.817376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:35.817376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:35.817376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:35.817376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:35.817376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:35.817376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.817376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:35.817376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.817672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:35.817672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:35.817672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:35.817672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.817672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:35.817672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:35.817672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:35.817672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:35.817672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.817672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:35.817672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:35.817672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:35.817672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.817904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:35.817904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:35.817904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:35.817904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:35.817904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:35.817904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:35.817904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:35.817904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:35.817904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:35.817904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:35.817904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:35.817904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:35.817904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:35.818126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.818126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:35.818126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.818126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:35.818126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:35.818126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:35.818126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:35.818126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:35.818126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.818126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:35.818126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:35.818126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.818126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:35.818126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:35.818126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.818441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:35.818441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:35.818441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.818441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:35.818441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:35.818441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:35.818441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:35.818441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.818441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:35.818441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:35.818441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:35.818441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.818441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:35.818441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:35.818441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:35.818441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:35.818441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:35.818441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:35.818441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:35.818762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:35.818762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:35.818762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:35.818762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:35.818762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:35.818762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:35.818762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.818762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:35.830698 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:35.831810 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:35.831810 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:35.831810 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:35.831810 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:35.832138 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:35.832393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.834648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:35.834648 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:35.834648 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:35.834648 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:35.834648 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:35.834648 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:35.834648 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.834648 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:35.834648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.834648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:35.834648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:35.834648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:35.834648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.834648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:35.834648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:35.835064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.835064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:35.835064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:35.835064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:35.835064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.835064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:35.835064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:35.835064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:35.835064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:35.835064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:35.835064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:35.835064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:35.835064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:35.835064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:35.835338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:35.835338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:35.835338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:35.835338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:35.835338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.835338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:35.835338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.835338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:35.835338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:35.835338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:35.835338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:35.835338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:35.835338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.835338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:35.835582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:35.836005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:35.836005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:35.836005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:35.836005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:35.836005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:35.836005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:35.836005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:35.836005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:35.836005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:35.836005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:38.705431 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:38.706010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:38.706010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:38.706283 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:38.733734 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:38.733939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:38.734431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:38.734431 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:38.734431 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:38.734431 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:38.734431 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:38.734431 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:38.734431 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:38.734431 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:38.735102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:38.735102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:38.735102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:38.735102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:38.735102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:38.735102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:38.735102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:38.735102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:38.735102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:38.735102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:38.735102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:38.735102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:38.735465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:38.735465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:38.735465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:38.735465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:38.735465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:38.735465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:38.735465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:38.735465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:38.735465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:38.735465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:38.735465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:38.735465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:38.735465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:38.735809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:38.735809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:38.735809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:38.735809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:38.735809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:38.735809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:38.735809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:38.735809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:38.735809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:38.735809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:38.735809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:38.735809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:38.735809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:38.735809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:38.735809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:38.736204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:38.736204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:38.736204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:38.736204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:38.736204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:38.736204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:38.736204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:38.736204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:38.736204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:38.736204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:38.736204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:38.736204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:38.736204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:38.736204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:38.736204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:38.736204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:38.736204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:38.736204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:38.736204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:38.736677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:38.736677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:38.736677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:38.736677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:38.736677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:38.736677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:40.773167 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:40.774480 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:40.774573 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:40.777834 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:40.778063 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:40.778238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:40.778421 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:40.778580 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:40.781719 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:40.782317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:40.782317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-27694673-a08c-41e6-8636-2613670955d3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:40.782718 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:40.782718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.786678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:40.786678 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:40.786678 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:40.786678 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:40.786678 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:40.786678 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:40.786678 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.786678 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:40.787704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.012 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.787704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:40.787704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:40.787704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:40.787704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.787704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:40.787704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:40.787704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.787704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:40.787704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:40.787704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:40.787704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.787942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:40.787942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:40.787942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:40.787942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:40.787942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:40.787942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:40.787942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:40.787942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:40.787942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:40.787942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:40.787942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:40.787942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:40.787942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:40.788149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.788149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:40.788799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.788799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:40.788799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:40.788799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:40.788799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:40.788799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:40.788799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.788799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:40.788799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:40.788799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.788799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:40.788799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:40.788799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.788799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:40.788799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:40.788799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.789077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:40.789077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:40.789077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.789077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:40.789077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:40.789077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:40.789077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.789077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:40.789077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:40.789077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:40.789077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:40.789077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:40.789077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:40.789077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:40.789077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:40.789077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:40.789077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:40.789077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:40.789077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:40.789407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:40.789407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.789407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:40.798602 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:40.799729 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:40.799824 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:40.799917 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:32:40.801033 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:40.802940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.804531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:40.804531 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:40.804531 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:40.804531 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:40.804531 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:40.804531 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:40.804531 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.804531 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:40.807009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.807009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:40.807009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:40.807009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:40.807009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.807009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:40.807009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:40.807009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:40.807009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.807009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:40.807009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:40.807009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:40.807009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:40.807009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:40.807009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:40.807347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:40.807347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:40.807347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:40.807347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:40.807347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:40.807347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:40.807347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:40.807347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.807347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:40.807347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.807347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:40.807347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:40.807347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:40.807347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.807579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:40.807579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:40.807579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:32:40.807579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:40.807579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:40.807579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.807579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:40.807579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:40.807579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:40.807579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.807579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:40.807579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:40.807579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:40.807796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:40.807796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:40.807796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:40.807796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:40.807796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:40.807796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:40.807796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:40.807796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:40.807796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:40.807796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:40.807796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.807796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:40.808005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.808005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:40.808005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:40.808005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:40.808005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:40.808005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:40.808005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.808005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:40.808005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:40.808005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.808005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:40.808005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:40.808005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.808005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:40.808005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:40.808005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.808363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:40.808363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:40.808363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:32:40.808363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:40.808363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:40.808363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.808363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:40.808363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:40.808363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:40.808363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.808363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:40.808363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:40.808363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:40.808363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:40.808363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:40.808363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:40.808363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:40.808363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:40.808879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:40.808879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:40.808879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:40.808879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:40.808879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:40.808879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.808879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:40.813561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:40.814083 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:40.814083 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:40.814083 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:40.814466 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:40.814660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.815120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:40.815120 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:40.815120 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:40.815120 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:40.815120 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:40.815120 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:40.815120 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.815120 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:40.816146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.816146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:40.816146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:40.816146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:40.816146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.816146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:40.816146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:40.816146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:40.816146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.816146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:40.816146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:40.816146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:40.816146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:40.816146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:40.816146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:40.816435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:40.816435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:40.816435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:40.816435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:40.816435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:40.816435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:40.816435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:40.816435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.816435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:40.818036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.818036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:40.818036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:40.818036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:40.818036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.818036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:40.818036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:40.818036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:40.818036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:40.818036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.818036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:40.818235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:40.818235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:40.818235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.818235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:40.818235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:40.818235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:40.818235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:40.818235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:40.818235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:40.818235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:40.818235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:40.818235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:40.818235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:40.818235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:40.818497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:40.818497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:40.818497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.818497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:40.818497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.818497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:40.818497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:40.818497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:40.818497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:40.818497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:40.818497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.818497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:40.818497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:40.818497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.818733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:40.818733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:40.818733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.818733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:40.818733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:40.818733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.818733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:40.818733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:40.818733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:40.818733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:40.818733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.818733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:40.818733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:40.818733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:40.818733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.818733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:40.818733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:40.818733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:40.818733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:40.819029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:40.819029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:40.819029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:40.819029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:40.819029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:40.819029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:40.819029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:40.819029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:40.819029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:40.819029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.819029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:40.831211 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:40.832055 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:40.832055 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:40.832055 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:40.835605 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:40.835798 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:40.836024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.837667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:40.837667 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:40.837667 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:40.837667 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:40.837667 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:40.837667 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:40.837667 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.837667 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:40.839410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.839410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:40.839410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:40.839410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:40.839410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.839410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:40.839410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:40.839410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.839410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:40.839410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:40.839410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:40.839410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.839625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:40.839625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:40.839625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:40.839625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:40.839625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:40.839625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:40.839625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:40.839625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:40.839625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:40.839625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:40.839625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:40.839625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:40.839625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:40.839833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.839833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:40.839833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.839833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:40.839833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:40.839833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:40.839833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:40.839833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:40.839833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.839833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:40.839833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:40.839833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.839833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:40.839833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:40.839833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.840137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:40.840137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:40.840137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:40.840137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:40.840137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:40.840137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.840137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:40.840137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:40.840137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:40.840137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:40.840137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:40.840137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:40.840137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:40.840137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:40.840137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:40.840137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:40.840137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:40.840137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:40.840137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:40.840480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:40.840480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:40.840480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:40.840480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:40.840480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:40.840480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:43.705799 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:43.707659 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:43.707659 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:43.707659 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:43.736439 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:43.736600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:43.737307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:43.737307 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:43.737307 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:43.737307 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:43.737307 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:43.737307 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:43.737307 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:43.737307 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:43.738022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:43.738022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:43.738022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:43.738022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:43.738022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:43.738022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:43.738022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:43.738022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:43.738022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:43.738022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:43.738022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:43.738022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:43.738421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:43.738421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:43.738421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:43.738421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:43.738421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:43.738421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:43.738421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:43.738421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:43.738421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:43.738421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:43.738421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:43.738421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:43.738421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:43.738768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:43.738768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:43.739734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:43.739734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:43.739734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:43.739734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:43.739734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:43.739734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:43.739734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:43.739734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:43.739734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:43.739734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:43.739734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:43.739734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:43.739734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:43.739734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:43.739734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:43.739734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:43.740989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:43.740989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:43.740989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:43.740989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:43.740989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:43.740989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:43.740989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:43.740989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:43.740989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:43.740989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:43.740989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:43.740989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:43.740989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:43.740989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:43.740989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:43.740989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:43.740989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:43.740989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:43.740989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:43.741598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:43.741598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:43.741598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:45.775007 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:45.776781 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:45.776781 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:45.776999 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:45.777184 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:45.777331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:45.777452 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:45.777559 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:45.777657 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:45.777752 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:45.777851 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-27694673-a08c-41e6-8636-2613670955d3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:45.777948 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:45.778065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.780617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:45.780617 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:45.780617 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:45.780617 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:45.780617 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:45.780617 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:45.780617 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.780617 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:45.780617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.780617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:45.780617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:45.780617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:45.780617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.780617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:45.780617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:45.781073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.781073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:45.781073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:45.781073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:45.781073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.781073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:45.781073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:45.781073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:45.781073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:45.781073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:45.781073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:45.781073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:45.781073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:45.781073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:45.781523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:45.781523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:45.781523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:45.781523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:45.781523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.781523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:45.781523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.781523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:45.781523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:45.781523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:45.781523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:45.781523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:45.781523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.781523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:45.781895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:45.782450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:45.782450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:45.782450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:45.782450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:45.782450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:45.782450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:45.782450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:45.782450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:45.782450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.782450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:45.800929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:45.801633 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:45.801633 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:45.801633 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:32:45.801862 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:45.802024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.802416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:45.802416 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:45.802416 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:45.802416 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:45.802416 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:45.802416 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:45.802416 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.802416 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:45.803352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.803352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:45.803352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:45.803352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:45.803352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.803352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:45.803352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:45.803352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:45.803352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.803352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:45.803352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:45.803352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:45.803352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:45.803352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:45.803352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:45.803919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:45.803919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:45.803919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:45.803919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:45.803919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:45.803919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:45.803919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:45.803919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.803919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:45.804846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.804846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:45.804846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:45.804846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:45.804846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.804846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:45.804846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:45.804846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:32:45.804846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:45.804846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:45.804846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.804846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:45.806280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:45.806280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:45.806280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.806280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:45.806280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:45.806280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:45.806280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:45.806280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:45.806280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:45.806280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:45.806280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:45.806280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:45.806280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:45.806280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:45.806755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:45.806755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:45.806755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.806755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:45.806755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.806755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:45.806755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:45.806755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:45.806755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:45.806755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:45.806755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.806755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:45.806755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:45.806755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.807230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:45.807230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:45.807230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.807230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:45.807230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:45.807230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.807230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:45.807230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:45.807230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:32:45.807230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:45.807230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:45.807230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.807230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:45.807230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:45.807230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:45.807230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.807230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:45.807230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:45.807230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:45.807558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:45.807558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:45.807558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:45.807558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:45.807558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:45.807558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:45.807558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:45.807558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:45.807558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:45.807558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:45.807558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.807558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:45.813845 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:45.814080 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:45.814242 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:45.814456 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:45.814983 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:45.815234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.817510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:45.817510 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:45.817510 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:45.817510 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:45.817510 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:45.817510 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:45.817510 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.817510 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:45.817510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.817510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:45.817510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:45.817510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:45.817510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.817510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:45.817510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:45.817510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:45.817510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.817510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:45.818131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:45.818131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:45.818131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:45.818131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:45.818131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:45.818131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:45.818131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:45.818131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:45.818131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:45.818131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:45.818131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:45.818131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:45.818131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.818131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:45.818131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.818811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:45.818811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:45.818811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:45.818811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.818811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:45.818811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:45.818811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:45.818811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:45.818811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.818811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:45.818811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:45.818811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:45.818811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.819674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:45.819674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:45.819674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:45.819674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:45.819674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:45.819674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:45.819674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:45.819674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:45.819674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:45.819674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:45.819674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:45.819674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:45.819674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:45.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:45.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:45.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:45.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:45.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:45.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:45.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:45.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:45.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:45.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:45.820117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.820537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:45.820537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:45.820537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.820537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:45.820537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:45.820537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:45.820537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:45.820537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.820537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:45.820537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:45.820537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:45.820537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.820537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:45.820537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:45.820537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:45.820537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:45.820537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:45.820537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:45.820537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:45.820932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:45.820932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:45.820932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:45.820932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:45.820932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:45.820932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:45.820932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.820932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:45.832559 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:45.833262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:45.833262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:45.833262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:45.834562 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:45.834759 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:45.835028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.835618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:45.835618 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:45.835618 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:45.835618 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:45.835618 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:45.835618 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:45.835618 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.835618 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:45.837398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.837398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:45.837398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:45.837398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:45.837398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.837398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:45.837398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:45.837398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.837398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:45.837398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:45.837398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:45.837398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.837638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:45.837638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:45.837638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:45.837638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:45.837638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:45.837638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:45.837638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:45.837638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:45.837638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:45.837638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:45.837638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:45.837638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:45.837638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:45.837842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.837842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:45.837842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.837842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:45.837842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:45.837842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:45.837842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:45.837842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:45.837842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.837842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:45.837842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:45.837842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.837842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:45.837842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:45.837842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.838091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:45.838091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:45.838091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:45.838091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:45.838091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:45.838091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.838091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:45.838091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:45.838091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:45.838091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:45.838091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:45.838091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:45.838091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:45.838091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:45.838091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:45.838091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:45.838091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:45.838091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:45.838091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:45.838488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:45.838488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:45.838488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:45.838488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:45.838488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:45.838488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:48.706339 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:48.707160 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:48.707160 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:48.707532 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:48.748708 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:48.748865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:48.749795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:48.749795 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:48.749795 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:48.749795 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:48.749795 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:48.749795 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:48.749795 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:48.749795 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:48.750571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:48.750571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:48.750571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:48.750571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:48.750571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:48.750571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:48.750571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:48.750571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:48.750571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:48.750571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:48.750571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:48.750571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:48.750810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:48.750810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:48.750810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:48.750810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:48.750810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:48.750810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:48.750810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:48.750810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:48.750810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:48.750810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:48.750810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:48.750810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:48.750810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:48.751045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:48.751045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:48.751414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:48.751414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:48.751414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:48.751414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:48.751414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:48.751414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:48.751414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:48.751414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:48.751414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:48.751414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:48.751414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:48.751414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:48.751414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:48.751414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:48.751414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:48.751414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:48.751883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:48.751883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:48.751883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:48.751883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:48.751883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:48.751883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:48.751883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:48.751883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:48.751883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:48.751883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:48.751883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:48.751883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:48.751883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:48.751883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:48.751883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:48.751883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:48.751883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:48.751883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:48.751883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:48.752687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:48.752687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:48.752687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:50.776538 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:50.777850 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:50.777850 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:50.779507 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:50.779665 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:50.779784 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:50.779880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:50.779990 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:50.780088 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:50.782319 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:50.782319 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-27694673-a08c-41e6-8636-2613670955d3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:50.782319 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:50.782319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.784504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:50.784504 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:50.784504 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:50.784504 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:50.784504 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:50.784504 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:50.784504 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.784504 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:50.786349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.786349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:50.786349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:50.786349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:50.786349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.786349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:50.786349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:50.786349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.786349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:50.786349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:50.786349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:50.786349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.786852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:50.786852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:50.786852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:50.786852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:50.786852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:50.786852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:50.786852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:50.786852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:50.786852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:50.786852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:50.786852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:50.786852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:50.786852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:50.787377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.787377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:50.787377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.787377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:50.787377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:50.787377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:50.787377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:50.787377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:50.787377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.787377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:50.787377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:50.787377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.787377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:50.787377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:50.787377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.787867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:50.787867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:50.787867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.787867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:50.787867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:50.787867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.787867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:50.787867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:50.787867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:50.787867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.787867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:50.787867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:50.787867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:50.787867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:50.787867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:50.787867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:50.787867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:50.787867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:50.787867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:50.788531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:50.788531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:50.788531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:50.788531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:50.788531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.788531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:50.805414 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:50.805414 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:50.805414 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:50.805414 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:32:50.806020 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:50.806020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.811092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:50.811092 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:50.811092 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:50.811092 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:50.811092 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:50.811092 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:50.811092 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.811092 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:50.811092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.811092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:50.811092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:50.811092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:50.811092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.811092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:50.811092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:50.811092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:50.811092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.811092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:50.811833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:50.811833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:50.811833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:50.811833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:50.811833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:50.811833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:50.811833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:50.811833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:50.811833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:50.811833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:50.811833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:50.811833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:50.811833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.811833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:50.813166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.813166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:50.813166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:50.813166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:50.813166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.813166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:50.813166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:50.813166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:32:50.813166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:50.813166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:50.813166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.813166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:50.814611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:50.814611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:50.814611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.814611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:50.814611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:50.814611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:50.814611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:50.814611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:50.814611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:50.814611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:50.814611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:50.814611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:50.814611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:50.814611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:50.815691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:50.815691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:50.815691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.815691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:50.815691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.815691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:50.815691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:50.815691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:50.815691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:50.815691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:50.815691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.815691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:50.815691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:50.815691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.815918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:50.815918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:50.815918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.815918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:50.815918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:50.815918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.815918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:50.815918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:50.815918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:32:50.815918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:50.815918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:50.815918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.815918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:50.815918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:50.815918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:50.815918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.815918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:50.815918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:50.815918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:50.816216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:50.816216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:50.816216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:50.816216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:50.816216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:50.816216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:50.816216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:50.816216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:50.816216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:50.816216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:50.816216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.816216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:50.816216 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:50.816216 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:50.816483 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:50.816483 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:50.818659 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:50.818659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.818659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:50.818659 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:50.818659 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:50.818659 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:50.818659 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:50.818659 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:50.818659 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.818659 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:50.819434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.819434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:50.819434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:50.819434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:50.819434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.819434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:50.819434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:50.819434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:50.819434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.819434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:50.819434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:50.819434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:50.819434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:50.819434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:50.819434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:50.819668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:50.819668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:50.819668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:50.819668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:50.819668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:50.819668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:50.819668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:50.819668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.819668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:50.819668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.819668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:50.819668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:50.819668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:50.819668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.819898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:50.819898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:50.819898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:50.819898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:50.819898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.819898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:50.819898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:50.819898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:50.819898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.819898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:50.819898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:50.819898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:50.819898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:50.820106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:50.820106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:50.820106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:50.820106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:50.820106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:50.820106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:50.820106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:50.820106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:50.820106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:50.820106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.820106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:50.820685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.820685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:50.820685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:50.820685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:50.820685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:50.820685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:50.820685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.820685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:50.820685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:50.820685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.820685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:50.820685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:50.820685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.820685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:50.820685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:50.820685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.820685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:50.821019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:50.821019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:50.821019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:50.821019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.821019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:50.821019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:50.821019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:50.821019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.821019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:50.821019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:50.821019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:50.821019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:50.821019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:50.821019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:50.821019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:50.821019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:50.821019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:50.821019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:50.821019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:50.821335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:50.821335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:50.821335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.821335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:50.833047 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:50.834128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:50.834225 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:50.834378 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:50.834870 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:50.835069 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:50.836279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.836279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:50.836279 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:50.836279 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:50.836279 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:50.836279 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:50.836279 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:50.836279 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.836279 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:50.836478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.836478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:50.836478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:50.836478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:50.836478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.836478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:50.836478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:50.836478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.836478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:50.836478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:50.836478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:50.836478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.836689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:50.836689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:50.836689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:50.836689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:50.836689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:50.836689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:50.836689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:50.836689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:50.836689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:50.836689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:50.836689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:50.836689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:50.836689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:50.836885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.836885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:50.837036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.837036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:50.837036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:50.837036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:50.837036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:50.837036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:50.837036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.837036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:50.837036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:50.837036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.837036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:50.837036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:50.837036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.837036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:50.837036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:50.837036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:50.837324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:50.837324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:50.837324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.837324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:50.837324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:50.837324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:50.837324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:50.837324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:50.837324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:50.837324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:50.837324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:50.837324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:50.837324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:50.837324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:50.837324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:50.837324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:50.837324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:50.837324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:50.837324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:50.837611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:50.837611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:50.837611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:53.706823 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:53.707349 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:53.707349 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:53.708043 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:53.721381 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=242 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:32:53.753182 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:53.754450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:53.756135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:53.756135 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:53.756135 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:53.756135 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:53.756135 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:53.756135 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:53.756135 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:53.756135 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:53.756135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:53.756135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:53.756135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:53.756135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:53.756135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:53.756135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:53.756135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:53.756586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:53.756586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:53.756586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:53.756586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:53.756586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:53.756586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:53.756586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:53.756586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:53.756586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:53.756586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:53.756586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:53.756586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:53.756586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:53.756586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:53.756835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:53.756835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:53.756835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:53.756835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:53.756835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:53.756835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:53.757744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:53.757744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:53.757744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:53.757744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:53.757744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:53.757744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:53.757744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:53.757744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:53.757744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:53.757744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:53.757744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:53.757744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:53.757744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:53.757744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:53.757744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:53.757744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:53.758112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:53.758112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:53.758112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:53.758112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:53.758112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:53.758112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:53.758112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:53.758112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:53.758112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:53.758112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:53.758112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:53.758112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:53.758112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:53.758112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:53.758112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:53.758112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:53.758112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:53.758112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:53.758112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:53.758472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:53.758472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:53.758472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:55.777053 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:55.777878 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:55.778021 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:55.779120 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:55.779397 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:55.779608 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:55.779753 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:55.779888 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:55.780031 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:55.780160 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-27694673-a08c-41e6-8636-2613670955d3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:55.780308 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:55.780493 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:55.781675 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:55.781817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.782476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:55.782476 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:55.782476 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:55.782476 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:55.782476 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:55.782476 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:55.782476 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.782476 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:55.783463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.783463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:55.783463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:55.783463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:55.783463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.783463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:55.783463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:55.783463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.783463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:55.783463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:55.783463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:55.783463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.783718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:55.783718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:55.783718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:55.783718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:55.783718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:55.783718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:55.783718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:55.783718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:55.783718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:55.783718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:55.783718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:55.783718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:55.783718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:55.783935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.783935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:55.784414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.784414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:55.784414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:55.784414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:55.784414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:55.784414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:55.784414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.784414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:55.784414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:55.784414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.784414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:55.784414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:55.784414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.784414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:55.784414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:55.784414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.784706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:32:55.784706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:55.784706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.784706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:55.784706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:55.784706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:55.784706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.784706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:55.784706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:55.784706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:55.784706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:55.784706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:55.784706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:55.784706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:55.784706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:55.784706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:55.784706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:55.784706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:55.784706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:55.785046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:55.785046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.785046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:55.806470 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:55.806913 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:55.807135 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:55.807135 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:32:55.812155 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:55.812155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.812155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:55.812155 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:55.812155 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:55.812155 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:55.812155 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:55.812155 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:55.812155 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.812155 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:55.812155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.812155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:55.812155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:55.812155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:55.812716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.812716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:55.812716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:55.812716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:55.812716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.812716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:55.812716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:55.812716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:55.812716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:55.812716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:55.812716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:55.812716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:55.812716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:55.812716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:55.812716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:55.812716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:55.812716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:55.812716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:55.813110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.813110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:55.813110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.813110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:55.813110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:55.813110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:55.813110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.813110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:55.813110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:55.813110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:32:55.813110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:55.813110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:55.813403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.813403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:55.813403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:55.813403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:55.813403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.813403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:55.813403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:55.813403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:55.813403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:55.813403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:55.813403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:55.813403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:55.813403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:55.813403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:55.814020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:55.814020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:55.814020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:55.814020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:55.814020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.814020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:55.814020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.814020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:55.814020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:55.814020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:55.814020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:55.814020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:55.814020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.814020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:55.814384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:55.814384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.814384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:55.814384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:55.814384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.814384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:55.814384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:55.814384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.814384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:32:55.814384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:32:55.814384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:32:55.814384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:55.814384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:55.814384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.814384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:55.814384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:55.814384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:55.814384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.814384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:55.814816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:55.814816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:55.814816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:55.814816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:55.814816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:55.814816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:55.814816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:55.814816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:55.814816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:55.814816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:55.814816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:55.814816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:55.814816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.814816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:55.816028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:55.816028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:55.816241 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:55.816404 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:55.817197 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:55.817197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.817504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:55.817504 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:55.817504 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:55.817504 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:55.817504 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:55.817504 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:55.817504 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.817504 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:55.818201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.818201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:32:55.818201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:55.818201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:55.818201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.818201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:55.818201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:55.818201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:32:55.818201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.818201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:32:55.818201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:55.818201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:32:55.818201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:55.818201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:32:55.818201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:32:55.818504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:55.818504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:32:55.818504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:55.818504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:32:55.818504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:32:55.818504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:55.818504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:32:55.818504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.818504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:32:55.819771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.819771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:55.819771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:55.819771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:55.819771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.819771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:55.819771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:55.819771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:55.819771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:55.819771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.819771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:55.820154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:55.820154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:55.820154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.820154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:55.820154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:55.820154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:55.820154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:55.820154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:55.820154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:55.820154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:55.820154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:55.820154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:55.820154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:55.820154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:55.820393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:55.820393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:55.820393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.820393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:55.821552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.821552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:55.821552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:55.821552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:55.821552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:55.821552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:55.821552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.821552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:55.821552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:55.821552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.821552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:55.821552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:55.821552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.821552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:55.821552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:55.821552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.821552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:32:55.821887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:32:55.821887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:32:55.821887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:32:55.821887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.821887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:55.821887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:55.821887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:55.821887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.821887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:55.821887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:55.821887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:55.821887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:55.821887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:55.821887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:55.821887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:55.821887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:55.821887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:55.821887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:55.821887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:55.822181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:55.822181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:55.822181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.822181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:55.834219 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:55.836071 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:55.836161 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:55.836373 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:55.836677 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:55.836928 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:55.837510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.837938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:55.837938 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:55.837938 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:55.837938 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:55.837938 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:55.837938 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:55.837938 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.837938 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:55.838537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.838537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:55.838537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:55.838537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:55.838537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.838537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:55.838537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:55.838537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.838537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:55.838537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:55.838537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:55.838537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.838789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:55.838789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:55.838789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:55.838789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:55.838789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:55.838789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:55.838789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:55.838789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:55.838789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:55.838789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:55.838789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:55.838789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:55.838789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:55.839044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.839044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:55.839175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.839175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:55.839175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:55.839175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:55.839175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:55.839175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:55.839175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.839175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:55.839175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:55.839175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.839175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:55.839175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:55.839175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.839175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:55.839175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:55.839175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:55.839587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:32:55.839587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:55.839587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.839587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:55.839587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:55.839587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:55.839587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:55.839587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:55.839587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:55.839587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:55.839587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:55.839587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:55.839587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:55.839587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:55.839587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:55.839587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:55.839587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:55.839587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:55.839587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:55.839875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:55.839875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:55.839875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:32:58.707205 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:32:58.707622 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:32:58.707622 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:32:58.708322 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:32:58.742806 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:32:58.743026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:58.743573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:32:58.743573 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:58.743573 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:32:58.743573 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:32:58.743573 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:58.743573 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:32:58.743573 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:58.743573 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:32:58.744363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:58.744363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:32:58.744363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:58.744363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:32:58.744363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:58.744363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:58.744363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:58.744363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:58.744363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:58.744363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:58.744363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:32:58.744363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:58.744774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:32:58.744774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:58.744774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:32:58.744774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:58.744774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:32:58.744774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:32:58.744774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:58.744774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:32:58.744774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:58.744774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:32:58.744774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:32:58.744774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:58.744774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:32:58.745165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:58.745165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:32:58.745165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:58.745165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:32:58.745165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:32:58.745165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:32:58.745165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:32:58.745165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:32:58.745165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:58.745165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:32:58.745165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:32:58.745165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:58.745165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:32:58.745165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:32:58.745165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:32:58.745570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:32:58.745570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:32:58.745570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:32:58.745570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:32:58.745570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:32:58.745570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:58.745570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:58.745570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:32:58.745570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:32:58.745570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:32:58.745570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:32:58.745570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:32:58.745570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:32:58.745570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:32:58.745570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:32:58.745570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:32:58.745570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:32:58.745570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:32:58.745570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:32:58.745946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:32:58.745946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:32:58.745946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:32:58.745946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:32:58.745946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:32:58.745946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:00.778025 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:00.778504 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:00.778504 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:00.780659 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:00.780659 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:00.780659 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:00.780659 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:00.780659 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:00.780821 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:00.780821 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-27694673-a08c-41e6-8636-2613670955d3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:00.780821 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:00.780821 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:00.780821 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:00.781026 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:00.781026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.781026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:00.781026 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:00.781026 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:00.781026 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:00.781026 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:00.781026 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:00.781026 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.781026 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:00.781443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.781443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:00.781443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:00.781443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:00.781443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.781443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:00.781443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:00.781443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.781443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:00.781443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:00.781443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:00.781443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.781809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:00.781809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:00.781809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:00.781809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:00.781809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:00.781809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:00.781809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:00.781809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:00.781809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:00.781809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:00.781809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:00.781809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:00.781809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:00.782195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.782195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:00.782195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.782195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:00.782195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:00.782195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:00.782195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:00.782195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:00.782195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.782195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:00.782195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:00.782195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.782195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:00.782195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:00.782195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.782731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:00.782731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:00.782731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.782731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:00.782731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:00.782731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.782731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:00.782731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:00.782731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:00.782731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.782731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:00.782731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:00.782731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:00.782731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:00.782731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:00.782731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:00.782731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:00.782731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:00.782731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:00.783175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:00.783175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:00.783175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:00.783175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:00.783175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.783175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:00.807042 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:00.807382 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:00.808561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:00.808561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:33:00.810652 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:00.810883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.811485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:00.811485 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:00.811485 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:00.811485 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:00.811485 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:00.811485 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:00.811485 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.811485 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:00.812688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.812688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:00.812688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:00.812688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:00.812688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.812688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:00.812688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:00.812688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:00.812688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.812688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:00.812688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:00.812688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:00.812688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:00.812688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:00.812688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:00.812951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:00.812951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:00.812951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:00.812951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:00.812951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:00.812951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:00.812951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:00.812951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.812951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:00.812951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.812951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:00.812951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:00.812951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:00.812951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.813226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:00.813226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:00.813226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:33:00.813226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:00.813226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:00.813226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.813226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:00.813226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:00.813226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:00.813226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.813226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:00.813226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:00.813226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:00.813443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:00.813443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:00.813443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:00.813443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:00.813443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:00.813443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:00.813443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:00.813443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:00.813443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:00.813443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:00.813443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.813443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:00.813946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.813946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:00.813946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:00.813946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:00.813946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:00.813946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:00.813946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.813946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:00.813946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:00.813946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.813946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:00.813946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:00.813946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.813946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:00.813946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:00.813946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.814221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:00.814221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:00.814221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:33:00.814221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:00.814221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:00.814221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.814221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:00.814221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:00.814221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:00.814221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.814221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:00.814221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:00.814221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:00.814221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:00.814221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:00.814221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:00.814221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:00.814221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:00.814622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:00.814622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:00.814622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:00.814622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:00.814622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:00.814622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.814622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:00.816095 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:00.816409 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:00.816997 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:00.816997 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:00.817238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:00.817729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.820705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:00.820705 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:00.820705 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:00.820705 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:00.820705 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:00.820705 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:00.820705 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.820705 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:00.821693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.821693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:00.821693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:00.821693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:00.821693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.821693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:00.821693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:00.821693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:00.821693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.821693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:00.821693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:00.821693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:00.821693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:00.821693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:00.821693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:00.822038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:00.822038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:00.822038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:00.822038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:00.822038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:00.822038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:00.822038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:00.822038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.822038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:00.823870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.823870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:00.823870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:00.823870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:00.823870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.823870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:00.823870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:00.823870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:00.823870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:00.823870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.823870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:00.824156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:00.824156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:00.824156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.824156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:00.824156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:00.824156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:00.824156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:00.824156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:00.824156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:00.824156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:00.824156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:00.824156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:00.824156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:00.824156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:00.824426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:00.824426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:00.824426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.824426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:00.826303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.826303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:00.826303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:00.826303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:00.826303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:00.826303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:00.826303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.826303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:00.826303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:00.826303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.826303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:00.826303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:00.826303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.826303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:00.826303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:00.826303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.826303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:00.826698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:00.826698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:00.826698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:00.826698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.826698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:00.826698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:00.826698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:00.826698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.826698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:00.826698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:00.826698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:00.826698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:00.826698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:00.826698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:00.826698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:00.826698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:00.826698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:00.826698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:00.826698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:00.827069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:00.827069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:00.827069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.827069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:00.839305 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:00.839480 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:00.839648 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:00.839718 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:00.841675 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:00.841958 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:00.842807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.850609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:00.850609 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:00.850609 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:00.850609 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:00.850609 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:00.850609 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:00.850609 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.850609 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:00.852461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.011 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.852461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:00.852461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:00.852461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:00.852461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.852461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:00.852461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:00.852461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.852461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:00.852461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:00.852461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:00.852461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.853111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:00.853111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:00.853111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:00.853111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:00.853111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:00.853111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:00.853111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:00.853111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:00.853111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:00.853111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:00.853111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:00.853111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:00.853111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:00.853492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.853492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:00.855866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.855866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:00.855866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:00.855866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:00.855866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:00.855866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:00.855866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.855866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:00.855866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:00.855866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.855866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:00.855866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:00.855866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.855866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:00.855866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:00.855866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:00.856170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:00.856170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:00.856170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.856170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:00.856170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:00.856170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:00.856170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:00.856170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:00.856170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:00.856170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:00.856170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:00.856170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:00.856170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:00.856170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:00.856170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:00.856170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:00.856170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:00.856170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:00.856170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:00.856510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:00.856510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:00.856510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:03.707659 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:03.708077 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:03.708077 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:03.708777 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:03.748854 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:03.749370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:03.749725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:03.749725 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:03.749725 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:03.749725 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:03.749725 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:03.749725 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:03.749725 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:03.749725 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:03.750438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:03.750438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:03.750438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:03.750438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:03.750438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:03.750438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:03.750438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:03.750438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:03.750438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:03.750438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:03.750438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:03.750438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:03.750764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:03.750764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:03.750764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:03.750764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:03.750764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:03.750764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:03.750764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:03.750764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:03.750764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:03.750764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:03.750764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:03.750764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:03.750764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:03.750960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:03.750960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:03.752297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:03.752297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:03.752297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:03.752297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:03.752297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:03.752297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:03.752297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:03.752297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:03.752297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:03.752297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:03.752297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:03.752297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:03.752297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:03.752297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:03.752297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:03.752297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:03.752740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:03.752740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:03.752740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:03.752740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:03.752740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:03.752740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:03.752740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:03.752740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:03.752740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:03.752740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:03.752740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:03.752740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:03.752740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:03.752740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:03.752740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:03.752740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:03.752740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:03.752740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:03.752740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:03.753210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:03.753210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:03.753210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:05.778403 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:05.778927 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:05.778927 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:05.779943 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:05.780140 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:05.780272 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:05.780409 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:05.780515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:05.780620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-27694673-a08c-41e6-8636-2613670955d3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:05.780736 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:05.780837 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:05.780966 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:05.781088 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:05.781188 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:05.781307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.781768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:05.781768 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:05.781768 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:05.781768 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:05.781768 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:05.781768 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:05.781768 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.781768 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:05.782518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.782518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:05.782518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:05.782518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:05.782518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.782518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:05.782518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:05.782518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.782518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:05.782518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:05.782518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:05.782518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.782761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:05.782761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:05.782761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:05.782761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:05.782761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:05.782761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:05.782761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:05.782761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:05.782761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:05.782761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:05.782761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:05.782761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:05.782761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:05.782968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.782968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:05.783261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.783261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:05.783261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:05.783261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:05.783261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:05.783261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:05.783261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.783261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:05.783261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:05.783261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.783261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:05.783261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:05.783261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.783261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:05.783261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:05.783261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.783549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:05.783549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:05.783549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.783549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:05.783549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:05.783549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:05.783549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.783549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:05.783549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:05.783549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:05.783549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:05.783549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:05.783549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:05.783549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:05.783549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:05.783549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:05.783549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:05.783549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:05.783549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:05.783845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:05.783845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.783845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:05.807550 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:05.807803 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:05.807877 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:05.807994 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:33:05.808355 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:05.808535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.808953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:05.808953 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:05.808953 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:05.808953 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:05.808953 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:05.808953 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:05.808953 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.808953 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:05.809502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.809502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:05.809502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:05.809502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:05.809502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.809502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:05.809502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:05.809502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:05.809502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.809502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:05.809502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:05.809502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:05.809502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:05.809502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:05.809502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:05.809765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:05.809765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:05.809765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:05.809765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:05.809765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:05.809765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:05.809765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:05.809765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.809765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:05.810123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.810123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:05.810123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:05.810123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:05.810123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.810123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:05.810123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:05.810123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:33:05.810123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:05.810123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:05.810123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.810123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:05.810737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:05.810737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:05.810737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.810737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:05.810737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:05.810737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:05.810737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:05.810737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:05.810737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:05.810737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:05.810737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:05.810737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:05.810737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:05.810737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:05.811048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:05.811048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:05.811048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.811048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:05.811048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.811048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:05.811048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:05.811048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:05.811048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:05.811048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:05.811048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.811048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:05.811048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:05.811048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.811281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:05.811281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:05.811281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.811281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:05.811281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:05.811281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.811281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:05.811281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:05.811281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:33:05.811281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:05.811281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:05.811281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.811281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:05.811281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:05.811281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:05.811281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.811281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:05.811281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:05.811281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:05.811573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:05.811573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:05.811573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:05.811573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:05.811573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:05.811573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:05.811573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:05.811573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:05.811573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:05.811573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:05.811573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.811573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:05.816455 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:05.816657 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:05.816741 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:05.816835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:05.817156 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:05.817365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.817762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:05.817762 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:05.817762 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:05.817762 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:05.817762 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:05.817762 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:05.817762 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.817762 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:05.818293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.818293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:05.818293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:05.818293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:05.818293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.818293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:05.818293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:05.818293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:05.818293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.818293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:05.818293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:05.818293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:05.818293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:05.818293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:05.818293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:05.818535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:05.818535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:05.818535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:05.818535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:05.818535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:05.818535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:05.818535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:05.818535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.818535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:05.818902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.818902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:05.818902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:05.818902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:05.818902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.818902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:05.818902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:05.818902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:05.818902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:05.818902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.818902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:05.819103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:05.819103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:05.819103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.819103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:05.819103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:05.819103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:05.819103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:05.819103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:05.819103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:05.819103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:05.819103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:05.819103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:05.819103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:05.819103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:05.819326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:05.819326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:05.819326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.819326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:05.819607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.819607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:05.819607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:05.819607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:05.819607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:05.819607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:05.819607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.819607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:05.819607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:05.819607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.819607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:05.819607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:05.819607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.819607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:05.819607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:05.819607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.819607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:05.820399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:05.820399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:05.820399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:05.820399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.820399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:05.820399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:05.820399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:05.820399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.820399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:05.820399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:05.820399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:05.820399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:05.820399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:05.820399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:05.820399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:05.820399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:05.820399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:05.820399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:05.820399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:05.820782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:05.820782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:05.820782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.820782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:05.839607 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:05.839827 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:05.839931 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:05.840064 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:05.840406 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:05.840603 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:05.840858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.841293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:05.841293 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:05.841293 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:05.841293 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:05.841293 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:05.841293 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:05.841293 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.841293 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:05.841822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.841822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:05.841822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:05.841822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:05.841822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.841822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:05.841822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:05.841822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.841822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:05.841822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:05.841822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:05.841822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.842076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:05.842076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:05.842076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:05.842076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:05.842076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:05.842076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:05.842076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:05.842076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:05.842076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:05.842076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:05.842076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:05.842076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:05.842076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:05.842345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.842345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:05.842741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.842741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:05.842741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:05.842741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:05.842741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:05.842741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:05.842741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.842741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:05.842741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:05.842741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.842741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:05.842741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:05.842741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.842741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:05.842741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:05.842741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:05.843656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:05.843656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:05.843656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.843656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:05.843656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:05.843656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:05.843656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:05.843656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:05.843656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:05.843656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:05.843656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:05.843656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:05.843656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:05.843656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:05.843656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:05.843656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:05.843656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:05.843656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:05.843656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:05.843929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:05.843929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:05.843929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:08.708040 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:08.710850 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:08.711064 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:08.712389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:08.749038 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:08.750880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:08.752334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:08.752334 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:08.752334 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:08.752334 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:08.752334 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:08.752334 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:08.752334 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:08.752334 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:08.753139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:08.753139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:08.753139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:08.753139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:08.753139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:08.753139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:08.753139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:08.753139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:08.753139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:08.753139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:08.753139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:08.753139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:08.753602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:08.753602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:08.753602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:08.753602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:08.753602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:08.753602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:08.753602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:08.753602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:08.753602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:08.753602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:08.753602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:08.753602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:08.753602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:08.753912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:08.753912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:08.753912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:08.753912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:08.753912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:08.753912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:08.753912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:08.753912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:08.753912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:08.753912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:08.753912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:08.753912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:08.753912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:08.753912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:08.753912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:08.754265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:08.754265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:08.754265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:08.754265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:08.754265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:08.754265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:08.754265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:08.754265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:08.754265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:08.754265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:08.754265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:08.754265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:08.754265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:08.754265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:08.754265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:08.754265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:08.754265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:08.754265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:08.754265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:08.754606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:08.754606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:08.754606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:08.754606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:08.754606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:08.754606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:10.778829 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:10.780729 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:10.780815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:10.781815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:10.781954 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:10.782073 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:10.782172 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:10.783226 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:10.783381 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-27694673-a08c-41e6-8636-2613670955d3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:10.783517 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:10.783624 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:10.783714 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:10.783813 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:10.783901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:10.784004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.784441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:10.784441 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:10.784441 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:10.784441 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:10.784441 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:10.784441 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:10.784441 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.784441 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:10.785132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.785132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:10.785132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:10.785132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:10.785132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.785132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:10.785132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:10.785132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.785132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:10.785132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:10.785132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:10.785132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.785442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:10.785442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:10.785442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:10.785442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:10.785442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:10.785442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:10.785442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:10.785442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:10.785442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:10.785442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:10.785442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:10.785442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:10.785442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:10.785669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.785669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:10.786081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.786081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:10.786081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:10.786081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:10.786081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:10.786081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:10.786081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.786081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:10.786081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:10.786081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.786081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:10.786081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:10.786081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.786081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:10.786081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:10.786081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:10.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:10.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:10.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:10.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:10.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:10.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:10.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:10.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:10.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:10.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:10.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:10.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:10.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:10.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:10.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:10.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:10.786772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:10.786772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.786772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:10.807990 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:10.808445 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:10.808556 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:10.808675 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:33:10.809046 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:10.809240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.810007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:10.810007 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:10.810007 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:10.810007 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:10.810007 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:10.810007 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:10.810007 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.810007 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:10.810572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.810572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:10.810572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:10.810572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:10.810572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.810572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:10.810572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:10.810572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:10.810572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.810572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:10.810572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:10.810572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:10.810572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:10.810572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:10.810572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:10.810889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:10.810889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:10.810889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:10.810889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:10.810889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:10.810889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:10.810889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:10.810889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.810889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:10.812396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.812396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:10.812396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:10.812396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:10.812396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.812396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:10.812396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:10.812396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:33:10.812396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:10.812396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:10.812396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.812396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:10.812655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:10.812655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:10.812655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.812655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:10.812655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:10.812655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:10.812655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:10.812655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:10.812655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:10.812655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:10.812655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:10.812655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:10.812655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:10.812655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:10.812916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:10.812916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:10.812916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.812916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:10.813527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.813527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:10.813527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:10.813527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:10.813527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:10.813527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:10.813527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.813527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:10.813527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:10.813527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.813527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:10.813527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:10.813527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.813527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:10.813527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:10.813527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.813816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:10.813816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:10.813816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:33:10.813816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:10.813816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:10.813816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.813816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:10.813816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:10.813816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:10.813816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.813816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:10.813816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:10.813816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:10.813816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:10.813816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:10.813816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:10.813816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:10.813816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:10.814182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:10.814182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:10.814182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:10.814182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:10.814182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:10.814182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.814182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:10.816767 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:10.817119 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:10.817205 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:10.817337 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:10.817616 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:10.817825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.818208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:10.818208 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:10.818208 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:10.818208 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:10.818208 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:10.818208 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:10.818208 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.818208 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:10.819115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.819115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:10.819115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:10.819115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:10.819115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.819115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:10.819115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:10.819115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:10.819115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.819115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:10.819115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:10.819115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:10.819115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:10.819115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:10.819115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:10.819444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:10.819444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:10.819444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:10.819444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:10.819444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:10.819444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:10.819444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:10.819444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.819444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:10.819444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.819444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:10.819444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:10.819444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:10.819444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.819733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:10.819733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:10.819733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:10.819733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:10.819733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.819733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:10.819733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:10.819733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:10.819733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.819733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:10.819733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:10.819733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:10.819733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:10.820009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:10.820009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:10.820009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:10.820009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:10.820009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:10.820009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:10.820009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:10.820009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:10.820009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:10.820009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.820009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:10.820196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.820196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:10.820196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:10.820196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:10.820196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:10.820196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:10.820196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.820196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:10.820196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:10.820196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.820196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:10.820196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:10.820196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.820196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:10.820196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:10.820196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.820196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:10.820625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:10.820625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:10.820625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:10.820625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.820625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:10.820625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:10.820625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:10.820625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.820625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:10.820625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:10.820625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:10.820625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:10.820625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:10.820625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:10.820625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:10.820625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:10.820625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:10.820625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:10.820625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:10.821033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:10.821033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:10.821033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.821033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:10.839961 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:10.840491 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:10.840491 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:10.840580 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:10.840794 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:10.840984 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:10.841218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.841625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:10.841625 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:10.841625 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:10.841625 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:10.841625 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:10.841625 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:10.841625 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.841625 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:10.842166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.842166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:10.842166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:10.842166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:10.842166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.842166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:10.842166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:10.842166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.842166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:10.842166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:10.842166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:10.842166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.842439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:10.842439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:10.842439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:10.842439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:10.842439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:10.842439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:10.842439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:10.842439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:10.842439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:10.842439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:10.842439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:10.842439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:10.842439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:10.842677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.842677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:10.842796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.842796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:10.842796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:10.842796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:10.842796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:10.842796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:10.842796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.842796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:10.842796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:10.842796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.842796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:10.842796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:10.842796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.842796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:10.842796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:10.842796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:10.843093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:10.843093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:10.843093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.843093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:10.843093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:10.843093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:10.843093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:10.843093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:10.843093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:10.843093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:10.843093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:10.843093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:10.843093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:10.843093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:10.843093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:10.843093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:10.843093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:10.843093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:10.843093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:10.843508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:10.843508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:10.843508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:13.710862 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:13.712026 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:13.712138 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:13.712783 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:14.010015 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:14.010208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:14.010783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:14.010783 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:14.010783 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:14.010783 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:14.010783 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:14.010783 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:14.010783 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:14.010783 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:14.011610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.299 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:14.011610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:14.011610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:14.011610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:14.011610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:14.011610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:14.011610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:14.011610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:14.011610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:14.011610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:14.011610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:14.011610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:14.012068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:14.012068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:14.012068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:14.012068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:14.012068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:14.012068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:14.012068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:14.012068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:14.012068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:14.012068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:14.012068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:14.012068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:14.012068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:14.012683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:14.012683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:14.012683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:14.012683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:14.012683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:14.012683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:14.012683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:14.012683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:14.012683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:14.012683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:14.012683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:14.012683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:14.012683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:14.012683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:14.012683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:14.012935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:14.012935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:14.012935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:14.012935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:14.012935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:14.012935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:14.012935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:14.012935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:14.012935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:14.012935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:14.012935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:14.012935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:14.012935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:14.012935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:14.012935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:14.012935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:14.012935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:14.012935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:14.012935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:14.013238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:14.013238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:14.013238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:14.013238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:14.013238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:14.013238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:15.781596 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:15.782239 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:15.782239 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:15.783994 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:15.785106 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:15.785389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:15.786529 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:15.786647 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-27694673-a08c-41e6-8636-2613670955d3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:15.786757 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:15.786862 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:15.786965 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:15.787470 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:15.787651 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:15.787773 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:15.787907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.788333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:15.788333 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:15.788333 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:15.788333 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:15.788333 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:15.788333 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:15.788333 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.788333 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:15.788960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.788960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:15.788960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:15.788960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:15.788960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.788960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:15.788960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:15.788960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.788960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:15.788960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:15.788960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:15.788960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.789372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:15.789372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:15.789372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:15.789372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:15.789372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:15.789372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:15.789372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:15.789372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:15.789372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:15.789372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:15.789372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:15.789372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:15.789372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:15.789718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.789718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:15.790799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.790799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:15.790799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:15.790799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:15.790799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:15.790799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:15.790799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.790799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:15.790799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:15.790799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.790799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:15.790799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:15.790799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.790799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:15.790799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:15.790799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.791106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:15.791106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:15.791106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.791106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:15.791106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:15.791106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:15.791106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.791106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:15.791106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:15.791106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:15.791106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:15.791106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:15.791106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:15.791106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:15.791106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:15.791106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:15.791106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:15.791106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:15.791106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:15.791461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:15.791461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.791461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:15.808445 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:15.808736 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:15.808914 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:15.809899 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:33:15.810235 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:15.810725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.812422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:15.812422 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:15.812422 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:15.812422 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:15.812422 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:15.812422 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:15.812422 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.812422 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:15.813100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.813100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:15.813100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:15.813100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:15.813100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.813100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:15.813100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:15.813100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:15.813100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.813100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:15.813100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:15.813100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:15.813100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:15.813100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:15.813100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:15.813408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:15.813408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:15.813408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:15.813408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:15.813408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:15.813408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:15.813408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:15.813408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.813408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:15.816769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.816769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:15.816769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:15.816769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:15.816769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.816769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:15.816769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:15.816769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:33:15.816769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:15.816769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:15.816769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.816769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:15.817046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:15.817046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:15.817046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.817046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:15.817046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:15.817046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:15.817046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:15.817046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:15.817046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:15.817046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:15.817046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:15.817046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:15.817046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:15.817046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:15.817282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:15.817282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:15.817282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.817282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:15.820087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.820087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:15.820087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:15.820087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:15.820087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:15.820087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:15.820087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.820087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:15.820087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:15.820087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.820087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:15.820087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:15.820087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.820087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:15.820087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:15.820087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.820415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:15.820415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:15.820415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:33:15.820415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:15.820415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:15.820415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.820415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:15.820415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:15.820415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:15.820415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.820415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:15.820415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:15.820415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:15.820415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:15.820415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:15.820415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:15.820415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:15.820415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:15.820817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:15.820817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:15.820817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:15.820817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:15.820817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:15.820817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.820817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:15.820817 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:15.822678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:15.822678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:15.822678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:15.822678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:15.823136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.823623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:15.823623 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:15.823623 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:15.823623 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:15.823623 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:15.823623 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:15.823623 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.823623 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:15.824525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.824525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:15.824525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:15.824525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:15.824525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.824525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:15.824525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:15.824525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:15.824525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.824525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:15.824525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:15.824525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:15.824525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:15.824525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:15.824525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:15.824785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:15.824785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:15.824785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:15.824785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:15.824785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:15.824785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:15.824785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:15.824785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.824785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:15.825945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.825945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:15.825945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:15.825945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:15.825945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.825945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:15.825945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:15.825945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:15.825945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:15.825945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.825945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:15.826283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:15.826283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:15.826283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.826283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:15.826283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:15.826283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:15.826283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:15.826283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:15.826283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:15.826283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:15.826283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:15.826283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:15.826283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:15.826283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:15.826560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:15.826560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:15.826560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.826560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:15.828841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.828841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:15.828841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:15.828841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:15.828841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:15.828841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:15.828841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.828841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:15.828841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:15.828841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.828841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:15.828841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:15.828841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.828841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:15.828841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:15.828841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.828841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:15.829290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:15.829290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:15.829290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:15.829290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.829290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:15.829290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:15.829290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:15.829290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.829290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:15.829290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:15.829290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:15.829290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:15.829290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:15.829290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:15.829290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:15.829290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:15.829290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:15.829290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:15.829290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:15.829619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:15.829619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:15.829619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.829619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:15.840357 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:15.840602 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:15.840699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:15.840845 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:15.841200 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:15.841493 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:15.841749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.842378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:15.842378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:15.842378 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:15.842378 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:15.842378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:15.842378 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:15.842378 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.842378 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:15.843231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.843231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:15.843231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:15.843231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:15.843231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.843231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:15.843231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:15.843231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.843231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:15.843231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:15.843231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:15.843231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.843476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:15.843476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:15.843476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:15.843476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:15.843476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:15.843476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:15.843476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:15.843476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:15.843476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:15.843476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:15.843476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:15.843476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:15.843476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:15.843682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.843682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:15.844204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.844204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:15.844204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:15.844204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:15.844204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:15.844204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:15.844204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.844204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:15.844204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:15.844204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.844204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:15.844204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:15.844204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.844204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:15.844204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:15.844204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:15.844505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:15.844505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:15.844505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.844505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:15.844505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:15.844505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:15.844505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:15.844505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:15.844505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:15.844505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:15.844505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:15.844505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:15.844505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:15.844505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:15.844505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:15.844505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:15.844505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:15.844505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:15.844505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:15.844790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:15.844790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:15.844790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:18.711416 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:18.712474 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:18.712474 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:18.712474 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:18.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:18.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:18.767021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:18.767021 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:18.767021 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:18.767021 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:18.767021 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:18.767021 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:18.767021 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:18.767021 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:18.767021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.054 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:18.767021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:18.767021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:18.767021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:18.767021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:18.767021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:18.767021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:18.767459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:18.767459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:18.767459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:18.767459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:18.767459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:18.767459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:18.767459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:18.767459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:18.767459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:18.767459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:18.767459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:18.767459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:18.767459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:18.767459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:18.767670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:18.767670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:18.767670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:18.767670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:18.767670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:18.767670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:18.767670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:18.767670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:18.767670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:18.767670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:18.767670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:18.767670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:18.767670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:18.767670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:18.767898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:18.768272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:18.768272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:18.768272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:18.768272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:18.768272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:18.768272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:18.768272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:18.768272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:18.768272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:20.783243 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:20.783243 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:20.783243 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:20.785335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:20.785335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:20.785335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:20.785335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:20.785335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-27694673-a08c-41e6-8636-2613670955d3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:20.785690 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:20.786456 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:20.786636 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:20.786824 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:20.786969 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:20.787547 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-ba8949a8-c177-40b2-aacd-094161feb4e3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:20.787726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.788312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:20.788312 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:20.788312 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:20.788312 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:20.788312 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:20.788312 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:20.788312 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.788312 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:20.789345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.789345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:20.789345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:20.789345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:20.789345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.789345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:20.789345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:20.789345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.789345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:20.789345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:20.789345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:20.789345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.789573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:20.789573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:20.789573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:20.789573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:20.789573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:20.789573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:20.789573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:20.789573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:20.789573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:20.789573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:20.789573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:20.789573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:20.789573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:20.789838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.789838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:20.790140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.790140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:20.790140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:20.790140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:20.790140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:20.790140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:20.790140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.790140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:20.790140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:20.790140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.790140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:20.790140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:20.790140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.790140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:20.790140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:20.790140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.790419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:20.790419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:20.790419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.790419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:20.790419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:20.790419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:20.790419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.790419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:20.790419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:20.790419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:20.790419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:20.790419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:20.790419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:20.790419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:20.790419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:20.790419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:20.790419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:20.790419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:20.790419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:20.790762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:20.790762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.790762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:20.808832 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:20.809770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:20.809770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:20.809770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:33:20.811377 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:20.811596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.812138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:20.812138 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:20.812138 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:20.812138 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:20.812138 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:20.812138 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:20.812138 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.812138 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:20.813136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.813136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:20.813136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:20.813136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:20.813136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.813136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:20.813136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:20.813136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:20.813136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.813136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:20.813136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:20.813136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:20.813136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:20.813136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:20.813136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:20.813397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:20.813397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:20.813397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:20.813397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:20.813397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:20.813397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:20.813397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:20.813397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.813397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:20.813984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.813984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:20.813984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:20.813984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:20.813984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.813984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:20.813984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:20.813984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:33:20.813984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:20.813984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:20.813984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.813984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:20.814232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:20.814232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:20.814232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.814232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:20.814232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:20.814232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:20.814232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:20.814232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:20.814232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:20.814232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:20.814232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:20.814232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:20.814232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:20.814232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:20.814448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:20.814448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:20.814448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.814448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:20.815097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.815097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:20.815097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:20.815097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:20.815097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:20.815097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:20.815097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.815097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:20.815097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:20.815097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.815097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:20.815097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:20.815097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.815097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:20.815097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:20.815097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.815418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:20.815418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:20.815418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:33:20.815418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:20.815418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:20.815418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.815418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:20.815418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:20.815418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:20.815418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.815418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:20.815418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:20.815418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:20.815418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:20.815418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:20.815418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:20.815418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:20.815418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:20.815668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:20.815668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:20.815668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:20.815668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:20.815668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:20.815668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.815668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:20.822167 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:20.822988 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:20.823137 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:20.823315 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:20.823680 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:20.823883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.824386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:20.824386 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:20.824386 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:20.824386 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:20.824386 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:20.824386 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:20.824386 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.824386 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:20.826282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.826282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:20.826282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:20.826282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:20.826282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.826282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:20.826282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:20.826282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:20.826282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.826282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:20.826282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:20.826282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:20.826282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:20.826282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:20.826282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:20.826570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:20.826570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:20.826570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:20.826570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:20.826570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:20.826570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:20.826570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:20.826570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.826570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:20.826570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.826570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:20.826570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:20.826570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:20.826570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.826869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:20.826869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:20.826869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:20.826869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:20.826869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.826869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:20.826869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:20.826869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:20.826869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.826869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:20.826869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:20.826869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:20.826869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:20.827100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:20.827100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:20.827100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:20.827100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:20.827100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:20.827100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:20.827100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:20.827100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:20.827100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:20.827100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.827100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:20.831308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.831308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:20.831308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:20.831308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:20.831308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:20.831308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:20.831308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.831308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:20.831308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:20.831308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.831308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:20.831308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:20.831308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.831308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:20.831308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:20.831308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.831308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:20.832117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:20.832117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:20.832117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.832117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:20.841011 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:20.842203 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:20.842381 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:20.842594 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:20.843039 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:20.843347 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:20.843787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.844442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:20.844442 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:20.844442 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:20.844442 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:20.844442 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:20.844442 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:20.844442 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.844442 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:20.845370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.845370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:20.845370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:20.845370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:20.845370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.845370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:20.845370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:20.845370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.845370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:20.845370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:20.845370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:20.845370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.845614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:20.845614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:20.845614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:20.845614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:20.845614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:20.845614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:20.845614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:20.845614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:20.845614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:20.845614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:20.845614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:20.845614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:20.845614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:20.845830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.845830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:20.846235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.846235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:20.846235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:20.846235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:20.846235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:20.846235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:20.846235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.846235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:20.846235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:20.846235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.846235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:20.846235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:20.846235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.846235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:20.846235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:20.846235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:20.846566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:20.846566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:20.846566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.846566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:20.846566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:20.846566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:20.846566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:20.846566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:20.846566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:20.846566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:20.846566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:20.846566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:20.846566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:20.846566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:20.846566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:20.846566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:20.846566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:20.846566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:20.846566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:20.846924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:20.846924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:20.846924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:23.711883 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:23.712360 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:23.712557 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:23.713219 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:23.755399 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:23.755777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:23.756485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:23.756485 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:23.756485 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:23.756485 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:23.756485 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:23.756485 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:23.756485 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:23.756485 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:23.757520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:23.757520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:23.757520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:23.757520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:23.757520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:23.757520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:23.757520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:23.757520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:23.757520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:23.757520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:23.757520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:23.757520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:23.757970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:23.757970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:23.757970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:23.757970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:23.757970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:23.757970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:23.757970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:23.757970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:23.757970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:23.757970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:23.757970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:23.757970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:23.757970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:23.758179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:23.758179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:23.759029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:23.759029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:23.759029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:23.759029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:23.759029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:23.759029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:23.759029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:23.759029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:23.759029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:23.759029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:23.759029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:23.759029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:23.759029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:23.759029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:23.759029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:23.759029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:23.759347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:23.759347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:23.759347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:23.759347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:23.759347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:23.759347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:23.759347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:23.759347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:23.759347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:23.759347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:23.759347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:23.759347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:23.759347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:23.759347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:23.759347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:23.759347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:23.759347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:23.759347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:23.759347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:23.759963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:23.759963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:23.759963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:25.782761 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:25.783283 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:25.785408 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:25.785408 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:25.785408 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:25.785408 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:25.785408 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:25.785573 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-27694673-a08c-41e6-8636-2613670955d3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:25.785573 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:25.785573 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:25.785573 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:25.785722 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:25.785894 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:25.786030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.786535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:25.786535 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:25.786535 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:25.786535 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:25.786535 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:25.786535 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:25.786535 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.786535 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:25.787330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.787330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:25.787330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:25.787330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:25.787330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.787330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:25.787330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:25.787330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.787330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:25.787330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:25.787330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:25.787330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.787612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:25.787612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:25.787612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:25.787612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:25.787612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:25.787612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:25.787612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:25.787612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:25.787612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:25.787612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:25.787612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:25.787612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:25.787612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:25.787864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.787864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:25.788087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.788087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:25.788087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:25.788087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:25.788087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:25.788087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:25.788087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.788087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:25.788087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:25.788087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.788087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:25.788087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:25.788087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.788087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:25.788087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:25.788087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.788440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:25.788440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:25.788440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.788440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:25.788440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:25.788440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:25.788440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.788440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:25.788440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:25.788440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:25.788440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:25.788440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:25.788440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:25.788440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:25.788440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:25.788440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:25.788440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:25.788440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:25.788440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:25.788769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:25.788769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.788769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:25.809342 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:25.809781 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:25.810016 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:25.810267 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:33:25.810770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:25.810993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.811670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:25.811670 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:25.811670 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:25.811670 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:25.811670 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:25.811670 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:25.811670 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.811670 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:25.812634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.812634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:25.812634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:25.812634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:25.812634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.812634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:25.812634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:25.812634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:25.812634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.812634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:25.812634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:25.812634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:25.812634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:25.812634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:25.812634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:25.812934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:25.812934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:25.812934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:25.812934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:25.812934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:25.812934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:25.812934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:25.812934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.812934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:25.813602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.813602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:25.813602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:25.813602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:25.813602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.813602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:25.813602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:25.813602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:33:25.813602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:25.813602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:25.813602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.813602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:25.813866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:25.813866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:25.813866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.813866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:25.813866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:25.813866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:25.813866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:25.813866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:25.813866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:25.813866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:25.813866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:25.813866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:25.813866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:25.813866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:25.814103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:25.814103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:25.814103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.814103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:25.814823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.814823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:25.814823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:25.814823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:25.814823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:25.814823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:25.814823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.814823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:25.814823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:25.814823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.814823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:25.814823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:25.814823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.814823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:25.814823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:25.814823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.815164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:25.815164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:25.815164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:33:25.815164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:25.815164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:25.815164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.815164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:25.815164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:25.815164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:25.815164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.815164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:25.815164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:25.815164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:25.815164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:25.815164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:25.815164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:25.815164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:25.815164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:25.815565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:25.815565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:25.815565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:25.815565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:25.815565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:25.815565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.815565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:25.822983 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:25.823348 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:25.823477 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:25.823584 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:25.824166 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:25.824217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.824700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:25.824700 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:25.824700 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:25.824700 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:25.824700 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:25.824700 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:25.824700 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.824700 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:25.825474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.825474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:25.825474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:25.825474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:25.825474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.825474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:25.825474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:25.825474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:25.825474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.825474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:25.825474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:25.825474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:25.825474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:25.825474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:25.825474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:25.825756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:25.825756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:25.825756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:25.825756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:25.825756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:25.825756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:25.825756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:25.825756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.825756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:25.826499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.826499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:25.826499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:25.826499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:25.826499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.826499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:25.826499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:25.826499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:25.826499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:25.826499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.826499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:25.826699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:25.826699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:25.826699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.826699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:25.826699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:25.826699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:25.826699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:25.826699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:25.826699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:25.826699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:25.826699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:25.826699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:25.826699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:25.826699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:25.826950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:25.826950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:25.826950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.826950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:25.827324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.827324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:25.827324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:25.827324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:25.827324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:25.827324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:25.827324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.827324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:25.827324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:25.827324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.827324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:25.827324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:25.827324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.827324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:25.827324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:25.827324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.827324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:25.827625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:25.827625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:25.827625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:25.827625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.827625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:25.827625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:25.827625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:25.827625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.827625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:25.827625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:25.827625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:25.827625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:25.827625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:25.827625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:25.827625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:25.827625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:25.827625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:25.827625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:25.827625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:25.827923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:25.827923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:25.827923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.827923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:25.842233 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:25.842573 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:25.842671 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:25.842822 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:25.843390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:25.843633 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:25.844030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.844710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:25.844710 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:25.844710 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:25.844710 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:25.844710 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:25.844710 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:25.844710 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.844710 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:25.845607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.845607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:25.845607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:25.845607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:25.845607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.845607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:25.845607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:25.845607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.845607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:25.845607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:25.845607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:25.845607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.845812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:25.845812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:25.845812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:25.845812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:25.845812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:25.845812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:25.845812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:25.845812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:25.845812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:25.845812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:25.845812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:25.845812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:25.845812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:25.846013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.846013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:25.846935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.846935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:25.846935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:25.846935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:25.846935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:25.846935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:25.846935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.846935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:25.846935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:25.846935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.846935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:25.846935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:25.846935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.846935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:25.846935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:25.846935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:25.847222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:25.847222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:25.847222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.847222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:25.847222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:25.847222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:25.847222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:25.847222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:25.847222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:25.847222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:25.847222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:25.847222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:25.847222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:25.847222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:25.847222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:25.847222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:25.847222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:25.847222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:25.847222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:25.847552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:25.847552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:25.847552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:28.712369 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:28.712782 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:28.712958 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:28.716965 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:28.794154 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:28.794658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:28.795017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:28.795017 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:28.795017 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:28.795017 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:28.795017 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:28.795017 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:28.795017 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:28.795017 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:28.795778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.082 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:28.795778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:28.795778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:28.795778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:28.795778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:28.795778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:28.795778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:28.795778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:28.795778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:28.795778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:28.795778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:28.795778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:28.796018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:28.796018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:28.796018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:28.796018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:28.796018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:28.796018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:28.796018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:28.796018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:28.796018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:28.796018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:28.796018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:28.796018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:28.796018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:28.796222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:28.796222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:28.796600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:28.796600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:28.796600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:28.796600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:28.796600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:28.796600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:28.796600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:28.796600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:28.796600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:28.796600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:28.796600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:28.796600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:28.796600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:28.796600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:28.796600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:28.796600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:28.796856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:28.796856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:28.796856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:28.796856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:28.796856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:28.796856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:28.796856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:28.796856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:28.796856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:28.796856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:28.796856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:28.796856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:28.796856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:28.796856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:28.796856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:28.796856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:28.796856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:28.796856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:28.796856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:28.797174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:28.797174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:28.797174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:30.785776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:30.787134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:30.788090 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:30.790499 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:30.790600 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:30.790751 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:30.790918 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:30.791340 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-27694673-a08c-41e6-8636-2613670955d3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:30.791340 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:30.791486 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:30.791558 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:30.791732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:30.791891 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:30.792188 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:30.792337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.793480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:30.793480 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:30.793480 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:30.793480 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:30.793480 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:30.793480 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:30.793480 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.793480 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:30.795555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.795555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:30.795555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:30.795555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:30.795555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.795555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:30.795555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:30.795555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.795555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:30.795555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:30.795555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:30.795555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.796222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:30.796222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:30.796222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:30.796222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:30.796222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:30.796222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:30.796222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:30.796222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:30.796222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:30.796222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:30.796222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:30.796222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:30.796222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:30.796830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.796830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:30.800048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.800048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:30.800048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:30.800048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:30.800048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:30.800048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:30.800048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.800048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:30.800048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:30.800048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.800048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:30.800048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:30.800048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.800048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:30.800048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:30.800048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.800904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:30.800904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:30.800904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.800904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:30.800904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:30.800904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:30.800904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.800904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:30.800904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:30.800904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:30.800904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:30.800904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:30.800904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:30.800904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:30.800904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:30.800904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:30.800904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:30.800904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:30.800904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:30.802159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:30.802159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.802159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:30.810163 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:30.811039 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:30.811039 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:30.811039 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:33:30.811213 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:30.811394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.814311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:30.814311 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:30.814311 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:30.814311 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:30.814311 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:30.814311 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:30.814311 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.814311 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:30.814311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.814311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:30.814311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:30.814311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:30.814311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.814311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:30.814311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:30.814311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:30.814311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.814311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:30.814749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:30.814749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:30.814749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:30.814749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:30.814749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:30.814749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:30.814749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:30.814749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:30.814749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:30.814749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:30.814749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:30.814749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:30.814749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.814749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:30.814749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.815047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:30.815047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:30.815047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:30.815047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.815047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:30.815047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:30.815047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:33:30.815047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:30.815047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:30.815047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.815047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:30.815047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:30.815047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:30.815284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.815284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:30.815284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:30.815284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:30.815284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:30.815284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:30.815284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:30.815284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:30.815284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:30.815284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:30.815284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:30.815284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:30.815284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:30.815284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:30.815522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.815522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:30.816042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.816042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:30.816042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:30.816042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:30.816042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:30.816042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:30.816042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.816042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:30.816042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:30.816042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.816042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:30.816042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:30.816042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.816042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:30.816042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:30.816042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.816525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:30.816525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:30.816525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:33:30.816525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:30.816525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:30.816525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.816525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:30.816525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:30.816525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:30.816525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.816525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:30.816525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:30.816525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:30.816525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:30.816525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:30.816525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:30.816525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:30.816525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:30.816860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:30.816860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:30.816860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:30.816860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:30.816860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:30.816860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.816860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:30.823503 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:30.823924 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:30.825005 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:30.825435 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:30.827361 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:30.827617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.828011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:30.828011 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:30.828011 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:30.828011 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:30.828011 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:30.828011 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:30.828011 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.828011 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:30.829209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.829209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:30.829209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:30.829209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:30.829209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.829209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:30.829209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:30.829209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:30.829209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.829209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:30.829209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:30.829209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:30.829209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:30.829209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:30.829209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:30.829842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:30.829842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:30.829842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:30.829842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:30.829842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:30.829842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:30.829842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:30.829842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.829842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:30.830901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.830901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:30.830901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:30.830901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:30.830901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.830901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:30.830901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:30.830901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:30.830901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:30.830901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.830901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:30.831171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:30.831171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:30.831171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.831171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:30.831171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:30.831171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:30.831171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:30.831171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:30.831171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:30.831171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:30.831171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:30.831171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:30.831171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:30.831171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:30.831418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:30.831418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:30.831418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.831418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:30.833445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.833445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:30.833445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:30.833445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:30.833445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:30.833445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:30.833445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.833445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:30.833445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:30.833445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.833445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:30.833445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:30.833445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.833445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:30.833445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:30.833445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.833445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:30.833785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:30.833785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:30.833785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:30.833785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.833785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:30.833785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:30.833785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:30.833785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.833785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:30.833785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:30.833785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:30.833785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:30.833785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:30.833785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:30.833785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:30.833785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:30.833785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:30.833785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:30.833785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:30.834115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:30.834115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:30.834115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.834115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:30.843754 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:30.844625 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:30.845509 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:30.845675 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:30.847265 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:30.847656 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:30.849195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.850541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:30.850541 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:30.850541 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:30.850541 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:30.850541 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:30.850541 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:30.850541 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.850541 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:30.853495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.853495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:30.853495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:30.853495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:30.853495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.853495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:30.853495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:30.853495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.853495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:30.853495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:30.853495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:30.853495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.853835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:30.853835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:30.853835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:30.853835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:30.853835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:30.853835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:30.853835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:30.853835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:30.853835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:30.853835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:30.853835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:30.853835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:30.853835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:30.854184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.854184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:30.857271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.857271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:30.857271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:30.857271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:30.857271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:30.857271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:30.857271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.857271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:30.857271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:30.857271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.857271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:30.857271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:30.857271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.857271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:30.857271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:30.857271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:30.858432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:30.858432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:30.858432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.858432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:30.858432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:30.858432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:30.858432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:30.858432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:30.858432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:30.858432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:30.858432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:30.858432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:30.858432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:30.858432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:30.858432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:30.858432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:30.858432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:30.858432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:30.858432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:30.858943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:30.858943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:30.858943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:33.714824 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:33.716069 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:33.716304 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:33.717464 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:33.799225 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:33.799513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:33.801134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:33.801134 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:33.801134 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:33.801134 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:33.801134 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:33.801134 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:33.801134 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:33.801134 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:33.801916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.085 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:33.801916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:33.801916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:33.801916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:33.801916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:33.801916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:33.801916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:33.801916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:33.801916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:33.801916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:33.801916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:33.801916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:33.802413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:33.802413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:33.802413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:33.802413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:33.802413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:33.802413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:33.802413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:33.802413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:33.802413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:33.802413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:33.802413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:33.802413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:33.802413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:33.802833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:33.802833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:33.802833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:33.802833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:33.802833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:33.802833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:33.802833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:33.802833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:33.802833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:33.802833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:33.802833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:33.802833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:33.802833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:33.802833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:33.802833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:33.803336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:33.803336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:33.803336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:33.803336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:33.803336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:33.803336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:33.803336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:33.803336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:33.803336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:33.803336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:33.803336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:33.803336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:33.803336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:33.803336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:33.803336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:33.803336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:33.803336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:33.803336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:33.803336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:33.803924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:33.803924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:33.803924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:33.803924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:33.803924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:33.803924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:35.787264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:35.788217 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:35.791528 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:35.793309 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:35.793665 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:35.793775 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:35.794186 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:35.794186 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-27694673-a08c-41e6-8636-2613670955d3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:35.794338 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:35.794515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:35.794664 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:35.794792 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:35.794920 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:35.795063 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:35.795443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.799368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:35.799368 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:35.799368 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:35.799368 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:35.799368 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:35.799368 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:35.799368 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.799368 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:35.803798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.803798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:35.803798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:35.803798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:35.803798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.803798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:35.803798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:35.803798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.803798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:35.803798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:35.803798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:35.803798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.804952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:35.804952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:35.804952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:35.804952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:35.804952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:35.804952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:35.804952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:35.804952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:35.804952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:35.804952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:35.804952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:35.804952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:35.804952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:35.805418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.805418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:35.808727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.808727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:35.808727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:35.808727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:35.808727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:35.808727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:35.808727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.808727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:35.808727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:35.808727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.808727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:35.808727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:35.808727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.808727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:35.808727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:35.808727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.809357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:35.809357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:35.809357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.809357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:35.809357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:35.809357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:35.809357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.809357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:35.809357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:35.809357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:35.809357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:35.809357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:35.809357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:35.809357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:35.809357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:35.809357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:35.809357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:35.809357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:35.809357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:35.809877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:35.809877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.809877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:35.810640 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:35.817361 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:35.817550 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:35.817699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:33:35.818151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:35.818826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.819412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:35.819412 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:35.819412 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:35.819412 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:35.819412 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:35.819412 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:35.819412 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.819412 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:35.820551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.820551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:35.820551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:35.820551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:35.820551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.820551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:35.820551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:35.820551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:35.820551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.820551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:35.820551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:35.820551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:35.820551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:35.820551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:35.820551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:35.821070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:35.821070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:35.821070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:35.821070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:35.821070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:35.821070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:35.821070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:35.821070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.821070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:35.824198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.824198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:35.824198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:35.824198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:35.824198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.824198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:35.824198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:35.824198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:33:35.824198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:35.824198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:35.824198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.824198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:35.824669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:35.824669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:35.824669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.824669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:35.824669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:35.824669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:35.824669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:35.824669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:35.824669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:35.824669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:35.824669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:35.824669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:35.824669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:35.824669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:35.825138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:35.825138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:35.825138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.825138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:35.825705 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:35.826983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.826983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:35.826983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:35.826983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:35.826983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:35.826983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:35.826983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.826983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:35.826983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:35.826983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.826983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:35.826983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:35.826983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.826983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:35.826983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:35.826983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.827606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:35.827606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:35.827606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:33:35.827606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:35.827606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:35.827606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.827606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:35.827606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:35.827606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:35.827606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.827606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:35.827606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:35.827606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:35.827606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:35.827606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:35.827606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:35.827606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:35.827606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:35.828154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:35.828154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:35.828154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:35.828154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:35.828154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:35.828154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.828154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:35.828154 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:35.828154 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:35.828154 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:35.829583 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:35.829880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.831376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:35.831376 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:35.831376 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:35.831376 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:35.831376 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:35.831376 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:35.831376 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.831376 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:35.832173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.832173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:35.832173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:35.832173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:35.832173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.832173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:35.832173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:35.832173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:35.832173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.832173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:35.832173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:35.832173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:35.832173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:35.832173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:35.832173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:35.832646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:35.832646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:35.832646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:35.832646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:35.832646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:35.832646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:35.832646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:35.832646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.832646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:35.833180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.833180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:35.833180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:35.833180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:35.833180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.833180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:35.833180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:35.833180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:35.833180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:35.833180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.833180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:35.833517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:35.833517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:35.833517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.833517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:35.833517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:35.833517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:35.833517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:35.833517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:35.833517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:35.833517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:35.833517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:35.833517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:35.833517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:35.833517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:35.833847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:35.833847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:35.833847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.833847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:35.834831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.834831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:35.834831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:35.834831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:35.834831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:35.834831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:35.834831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.834831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:35.834831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:35.834831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.834831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:35.834831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:35.834831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.834831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:35.834831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:35.834831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.834831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:35.835341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:35.835341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:35.835341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:35.835341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.835341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:35.835341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:35.835341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:35.835341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.835341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:35.835341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:35.835341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:35.835341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:35.835341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:35.835341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:35.835341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:35.835341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:35.835341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:35.835341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:35.835341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:35.835792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:35.835792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:35.835792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.835792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:35.845067 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:35.845864 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:35.845951 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:35.846151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:35.847161 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:35.847538 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:35.848463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:35.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:35.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:35.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:35.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:35.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:35.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:35.850297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.850297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:35.850297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:35.850297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:35.850297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.850297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:35.850297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:35.850297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.850297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:35.850297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:35.850297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:35.850297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.850701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:35.850701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:35.850701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:35.850701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:35.850701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:35.850701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:35.850701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:35.850701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:35.850701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:35.850701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:35.850701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:35.850701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:35.850701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:35.851082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.851082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:35.851450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.851450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:35.851450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:35.851450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:35.851450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:35.851450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:35.851450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.851450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:35.851450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:35.851450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.851450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:35.851450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:35.851450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.851450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:35.851450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:35.851450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:35.851936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:35.851936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:35.851936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.851936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:35.851936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:35.851936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:35.851936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:35.851936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:35.851936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:35.851936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:35.851936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:35.851936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:35.851936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:35.851936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:35.851936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:35.851936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:35.851936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:35.851936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:35.851936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:35.852592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:35.852592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:35.852592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:38.716527 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:38.716899 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:38.716933 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:38.717711 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:38.769572 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:38.769750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:38.770322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:38.770322 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:38.770322 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:38.770322 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:38.770322 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:38.770322 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:38.770322 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:38.770322 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:38.771032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.053 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:38.771032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:38.771032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:38.771032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:38.771032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:38.771032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:38.771032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:38.771032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:38.771032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:38.771032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:38.771032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:38.771032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:38.771309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:38.771309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:38.771309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:38.771309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:38.771309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:38.771309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:38.771309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:38.771309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:38.771309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:38.771309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:38.771309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:38.771309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:38.771309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:38.771557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:38.771557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:38.771772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:38.771772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:38.771772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:38.771772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:38.771772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:38.771772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:38.771772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:38.771772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:38.771772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:38.771772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:38.771772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:38.771772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:38.771772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:38.771772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:38.771772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:38.771772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:38.772106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:38.772106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:38.772106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:38.772106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:38.772106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:38.772106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:38.772106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:38.772106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:38.772106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:38.772106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:38.772106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:38.772106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:38.772106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:38.772106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:38.772106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:38.772106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:38.772106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:38.772106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:38.772106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:38.772485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:38.772485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:38.772485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:40.788089 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:40.788708 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:40.788791 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:40.791417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:40.791417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:40.791417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:40.791417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:40.791797 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-27694673-a08c-41e6-8636-2613670955d3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:40.791797 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:40.791797 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:40.792012 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:40.792082 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:40.792207 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:40.792377 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:40.792587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.793558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:40.793558 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:40.793558 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:40.793558 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:40.793558 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:40.793558 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:40.793558 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.793558 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:40.794614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.794614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:40.794614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:40.794614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:40.794614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.794614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:40.794614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:40.794614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.794614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:40.794614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:40.794614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:40.794614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.795289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:40.795289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:40.795289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:40.795289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:40.795289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:40.795289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:40.795289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:40.795289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:40.795289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:40.795289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:40.795289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:40.795289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:40.795289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:40.795937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.795937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:40.795937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.795937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:40.795937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:40.795937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:40.795937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:40.795937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:40.795937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.795937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:40.795937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:40.795937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.795937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:40.795937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:40.795937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.796789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:40.796789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:40.796789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.796789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:40.796789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:40.796789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.796789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:40.796789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:40.796789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:40.796789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.796789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:40.796789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:40.796789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:40.796789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:40.796789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:40.796789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:40.796789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:40.796789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:40.796789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:40.797771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:40.797771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:40.797771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:40.797771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:40.797771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.797771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:40.817476 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:40.817785 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:40.817891 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:40.817963 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:33:40.818271 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:40.818439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.818821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:40.818821 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:40.818821 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:40.818821 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:40.818821 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:40.818821 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:40.818821 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.818821 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:40.819311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.819311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:40.819311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:40.819311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:40.819311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.819311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:40.819311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:40.819311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:40.819311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.819311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:40.819311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:40.819311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:40.819311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:40.819311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:40.819311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:40.819654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:40.819654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:40.819654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:40.819654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:40.819654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:40.819654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:40.819654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:40.819654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.819654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:40.819877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.819877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:40.819877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:40.819877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:40.819877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.819877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:40.819877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:40.819877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:33:40.819877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:40.819877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:40.819877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.819877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:40.820177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:40.820177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:40.820177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.820177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:40.820177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:40.820177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:40.820177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:40.820177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:40.820177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:40.820177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:40.820177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:40.820177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:40.820177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:40.820177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:40.820574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:40.820574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:40.820574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.820574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:40.820574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.820574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:40.820574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:40.820574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:40.820574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:40.820574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:40.820574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.820574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:40.820574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:40.820574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.820914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:40.820914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:40.820914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.820914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:40.820914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:40.820914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.820914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:40.820914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:40.820914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:33:40.820914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:40.820914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:40.820914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.820914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:40.820914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:40.820914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:40.820914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.820914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:40.820914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:40.820914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:40.821368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:40.821368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:40.821368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:40.821368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:40.821368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:40.821368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:40.821368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:40.821368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:40.821368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:40.821368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:40.821368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.821368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:40.827453 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:40.829247 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:40.829247 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:40.829247 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:40.830006 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:40.830240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.830868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:40.830868 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:40.830868 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:40.830868 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:40.830868 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:40.830868 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:40.830868 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.830868 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:40.831651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.831651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:40.831651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:40.831651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:40.831651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.831651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:40.831651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:40.831651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:40.831651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.831651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:40.831651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:40.831651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:40.831651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:40.831651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:40.831651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:40.832488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:40.832488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:40.832488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:40.832488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:40.832488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:40.832488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:40.832488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:40.832488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.832488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:40.834362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.834362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:40.834362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:40.834362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:40.834362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.834362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:40.834362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:40.834362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:40.834362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:40.834362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.834362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:40.834995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:40.834995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:40.834995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.834995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:40.834995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:40.834995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:40.834995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:40.834995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:40.834995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:40.834995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:40.834995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:40.834995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:40.834995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:40.834995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:40.835423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:40.835423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:40.835423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.835423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:40.835423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.835423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:40.835423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:40.835423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:40.835423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:40.835423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:40.835423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.835423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:40.835423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:40.835423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.835791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:40.835791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:40.835791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.835791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:40.835791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:40.835791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.835791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:40.835791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:40.835791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:40.835791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:40.835791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.835791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:40.835791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:40.835791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:40.835791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.835791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:40.835791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:40.835791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:40.835791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:40.836498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:40.836498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:40.836498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:40.836498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:40.836498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:40.836498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:40.836498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:40.836498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:40.836498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:40.836498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.836498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:40.845935 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:40.849762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:40.849929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:40.850152 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:40.851934 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:40.852241 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:40.852651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.853324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:40.853324 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:40.853324 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:40.853324 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:40.853324 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:40.853324 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:40.853324 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.853324 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:40.854350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.854350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:40.854350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:40.854350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:40.854350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.854350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:40.854350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:40.854350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.854350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:40.854350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:40.854350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:40.854350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.854793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:40.854793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:40.854793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:40.854793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:40.854793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:40.854793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:40.854793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:40.854793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:40.854793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:40.854793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:40.854793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:40.854793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:40.854793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:40.855180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.855180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:40.855776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.855776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:40.855776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:40.855776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:40.855776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:40.855776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:40.855776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.855776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:40.855776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:40.855776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.855776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:40.855776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:40.855776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.855776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:40.855776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:40.855776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:40.856414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:40.856414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:40.856414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.856414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:40.856414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:40.856414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:40.856414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:40.856414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:40.856414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:40.856414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:40.856414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:40.856414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:40.856414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:40.856414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:40.856414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:40.856414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:40.856414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:40.856414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:40.856414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:40.856916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:40.856916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:40.856916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:43.716960 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:43.717636 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:43.717776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:43.718734 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:43.771461 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:43.771863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:43.772571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:43.772571 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:43.772571 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:43.772571 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:43.772571 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:43.772571 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:43.772571 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:43.772571 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:43.775960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:43.775960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:43.775960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:43.775960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:43.775960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:43.775960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:43.775960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:43.775960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:43.775960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:43.775960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:43.775960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:43.775960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:43.776468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:43.776468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:43.776468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:43.776468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:43.776468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:43.776468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:43.776468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:43.776468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:43.776468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:43.776468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:43.776468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:43.776468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:43.776468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:43.776808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:43.776808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:43.776808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:43.776808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:43.776808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:43.776808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:43.776808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:43.776808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:43.776808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:43.776808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:43.776808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:43.776808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:43.776808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:43.776808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:43.776808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:43.777178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:43.777178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:43.777178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:43.777178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:43.777178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:43.777178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:43.777178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:43.777178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:43.777178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:43.777178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:43.777178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:43.777178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:43.777178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:43.777178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:43.777178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:43.777178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:43.777178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:43.777178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:43.777178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:43.777648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:43.777648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:43.777648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:43.777648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:43.777648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:43.777648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:45.788742 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:45.789142 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:45.789142 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:45.790564 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:45.790750 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:45.790865 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:45.791029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:45.791134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-27694673-a08c-41e6-8636-2613670955d3, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:45.791237 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:45.791359 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:45.791461 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:45.791558 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:45.791653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:45.791752 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:45.791929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.792411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:45.792411 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:45.792411 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:45.792411 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:45.792411 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:45.792411 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:45.792411 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.792411 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:45.793499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.793499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:45.793499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:45.793499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:45.793499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.793499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:45.793499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:45.793499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.793499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:45.793499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:45.793499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:45.793499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.793743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:45.793743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:45.793743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:45.793743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:45.793743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:45.793743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:45.793743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:45.793743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:45.793743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:45.793743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:45.793743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:45.793743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:45.793743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:45.793948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.793948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:45.794655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.794655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:45.794655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:45.794655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:45.794655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:45.794655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:45.794655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.794655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:45.794655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:45.794655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.794655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:45.794655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:45.794655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.794655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:45.794655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:45.794655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.794986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:45.794986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:45.794986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.794986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:45.794986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:45.794986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:45.794986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.794986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:45.794986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:45.794986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:45.794986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:45.794986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:45.794986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:45.794986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:45.794986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:45.794986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:45.794986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:45.794986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:45.794986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:45.795317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:45.795317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.795317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:45.817843 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:45.818125 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:45.818264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:45.818425 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:33:45.818792 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:45.819030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.819522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:45.819522 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:45.819522 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:45.819522 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:45.819522 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:45.819522 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:45.819522 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.819522 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:45.820396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.820396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:45.820396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:45.820396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:45.820396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.820396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:45.820396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:45.820396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:45.820396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.820396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:45.820396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:45.820396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:45.820396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:45.820396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:45.820396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:45.820766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:45.820766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:45.820766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:45.820766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:45.820766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:45.820766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:45.820766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:45.820766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.820766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:45.821323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.821323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:45.821323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:45.821323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:45.821323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.821323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:45.821323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:45.821323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:33:45.821323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:45.821323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:45.821323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.821323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:45.821588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:45.821588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:45.821588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.821588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:45.821588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:45.821588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:45.821588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:45.821588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:45.821588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:45.821588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:45.821588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:45.821588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:45.821588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:45.821588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:45.821824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:45.821824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:45.821824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.821824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:45.822548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.822548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:45.822548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:45.822548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:45.822548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:45.822548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:45.822548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.822548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:45.822548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:45.822548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.822548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:45.822548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:45.822548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.822548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:45.822548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:45.822548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.822876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:45.822876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:45.822876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:33:45.822876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:45.822876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:45.822876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.822876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:45.822876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:45.822876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:45.822876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.822876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:45.822876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:45.822876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:45.822876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:45.822876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:45.822876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:45.822876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:45.822876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:45.823203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:45.823203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:45.823203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:45.823203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:45.823203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:45.823203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.823203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:45.827859 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:45.828104 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:45.828234 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:45.828394 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:45.828688 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:45.828915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.829380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:45.829380 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:45.829380 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:45.829380 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:45.829380 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:45.829380 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:45.829380 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.829380 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:45.830227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.830227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:45.830227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:45.830227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:45.830227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.830227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:45.830227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:45.830227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:45.830227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.830227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:45.830227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:45.830227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:45.830227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:45.830227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:45.830227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:45.830487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:45.830487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:45.830487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:45.830487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:45.830487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:45.830487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:45.830487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:45.830487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.830487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:45.831085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.831085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:45.831085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:45.831085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:45.831085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.831085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:45.831085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:45.831085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:45.831085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:45.831085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.831085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:45.831327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:45.831327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:45.831327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.831327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:45.831327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:45.831327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:45.831327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:45.831327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:45.831327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:45.831327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:45.831327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:45.831327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:45.831327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:45.831327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:45.831570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:45.831570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:45.831570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.831570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:45.832362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.832362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:45.832362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:45.832362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:45.832362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:45.832362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:45.832362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.832362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:45.832362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:45.832362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.832362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:45.832362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:45.832362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.832362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:45.832362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:45.832362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.832362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:45.832684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:45.832684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:45.832684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:45.832684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.832684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:45.832684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:45.832684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:45.832684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.832684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:45.832684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:45.832684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:45.832684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:45.832684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:45.832684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:45.832684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:45.832684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:45.832684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:45.832684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:45.832684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:45.833083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:45.833083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:45.833083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.833083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:45.849763 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:45.850039 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:45.850168 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:45.850374 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:45.850650 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:45.850886 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:45.851198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.851653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:45.851653 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:45.851653 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:45.851653 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:45.851653 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:45.851653 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:45.851653 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.851653 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:45.852594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.852594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:45.852594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:45.852594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:45.852594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.852594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:45.852594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:45.852594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.852594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:45.852594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:45.852594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:45.852594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.852850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:45.852850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:45.852850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:45.852850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:45.852850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:45.852850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:45.852850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:45.852850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:45.852850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:45.852850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:45.852850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:45.852850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:45.852850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:45.853100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.853100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:45.853678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.853678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:45.853678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:45.853678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:45.853678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:45.853678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:45.853678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.853678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:45.853678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:45.853678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.853678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:45.853678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:45.853678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.853678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:45.853678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:45.853678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:45.853991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:45.853991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:45.853991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.853991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:45.853991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:45.853991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:45.853991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:45.853991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:45.853991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:45.853991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:45.853991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:45.853991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:45.853991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:45.853991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:45.853991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:45.853991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:45.853991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:45.853991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:45.853991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:45.854431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:45.854431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:45.854431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:48.717487 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:48.717901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:48.717901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:48.718720 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:48.754389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:48.754595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:48.755105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:48.755105 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:48.755105 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:48.755105 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:48.755105 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:48.755105 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:48.755105 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:48.755105 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:48.755805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:48.755805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:48.755805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:48.755805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:48.755805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:48.755805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:48.755805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:48.755805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:48.755805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:48.755805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:48.755805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:48.755805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:48.756041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:48.756041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:48.756041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:48.756041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:48.756041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:48.756041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:48.756041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:48.756041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:48.756041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:48.756041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:48.756041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:48.756041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:48.756041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:48.756230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:48.756230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:48.756625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:48.756625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:48.756625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:48.756625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:48.756625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:48.756625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:48.756625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:48.756625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:48.756625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:48.756625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:48.756625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:48.756625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:48.756625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:48.756625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:48.756625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:48.756625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:48.756876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:48.756876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:48.756876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:48.756876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:48.756876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:48.756876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:48.756876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:48.756876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:48.756876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:48.756876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:48.756876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:48.756876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:48.756876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:48.756876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:48.756876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:48.756876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:48.756876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:48.756876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:48.756876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:48.757166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:48.757166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:48.757166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:50.789485 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:50.790061 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:50.790184 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:50.791518 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:50.791722 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:50.791857 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:50.791997 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:50.792113 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:50.792212 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:50.792341 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:50.792439 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:50.792551 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:50.792650 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:50.792758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.793543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:50.793543 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:50.793543 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:50.793543 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:50.793543 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:50.793543 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:50.793543 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.793543 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:50.794485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.794485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:50.794485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:50.794485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:50.794485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.794485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:50.794485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:50.794485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.794485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:50.794485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:50.794485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:50.794485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.794685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:50.794685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:50.794685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:50.794685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:50.794685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:50.794685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:50.794685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:50.794685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:50.794685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:50.794685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:50.794685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:50.794685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:50.794685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:50.794920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.794920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:50.795186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.795186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:50.795186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:50.795186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:50.795186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:50.795186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:50.795186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.795186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:50.795186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:50.795186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.795186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:50.795186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:50.795186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.795186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:50.795186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:50.795186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.795485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:50.795485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:50.795485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.795485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:50.795485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:50.795485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:50.795485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.795485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:50.795485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:50.795485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:50.795485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:50.795485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:50.795485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:50.795485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:50.795485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:50.795485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:50.795485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:50.795485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:50.795485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:50.795796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:50.795796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.795796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:50.821482 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:50.821850 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:50.821944 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:50.822047 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:33:50.825810 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:50.826068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.826635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:50.826635 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:50.826635 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:50.826635 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:50.826635 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:50.826635 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:50.826635 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.826635 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:50.827271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.827271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:50.827271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:50.827271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:50.827271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.827271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:50.827271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:50.827271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:50.827271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.827271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:50.827271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:50.827271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:50.827271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:50.827271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:50.827271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:50.827564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:50.827564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:50.827564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:50.827564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:50.827564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:50.827564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:50.827564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:50.827564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.827564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:50.827768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.827768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:50.827768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:50.827768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:50.827768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.827768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:50.827768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:50.827768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:33:50.827768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:50.827768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:50.827768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.827768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:50.828036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:50.828036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:50.828036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.828036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:50.828036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:50.828036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:50.828036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:50.828036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:50.828036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:50.828036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:50.828036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:50.828036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:50.828036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:50.828036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:50.828295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:50.828295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:50.828295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.828295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:50.828680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.828680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:50.828680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:50.828680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:50.828680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:50.828680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:50.828680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.828680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:50.828680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:50.828680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.828680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:50.828680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:50.828680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.828680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:50.828680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:50.828680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.828949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:50.828949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:50.828949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:33:50.828949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:50.828949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:50.828949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.828949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:50.828949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:50.828949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:50.828949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.828949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:50.828949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:50.828949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:50.828949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:50.828949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:50.828949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:50.828949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:50.828949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:50.829298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:50.829298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:50.829298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:50.829298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:50.829298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:50.829298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.829298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:50.829427 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:50.829638 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:50.829709 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:50.829800 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:50.830219 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:50.830498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.831290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:50.831290 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:50.831290 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:50.831290 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:50.831290 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:50.831290 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:50.831290 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.831290 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:50.831704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.831704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:50.831704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:50.831704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:50.831704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.831704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:50.831704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:50.831704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:50.831704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.831704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:50.831704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:50.831704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:50.831704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:50.831704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:50.831704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:50.832028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:50.832028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:50.832028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:50.832028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:50.832028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:50.832028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:50.832028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:50.832028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.832028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:50.832272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.832272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:50.832272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:50.832272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:50.832272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.832272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:50.832272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:50.832272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:50.832272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:50.832272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.832272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:50.832481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:50.832481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:50.832481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.832481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:50.832481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:50.832481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:50.832481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:50.832481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:50.832481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:50.832481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:50.832481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:50.832481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:50.832481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:50.832481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:50.832726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:50.832726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:50.832726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.832726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:50.832936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.832936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:50.832936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:50.832936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:50.832936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:50.832936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:50.832936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.832936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:50.832936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:50.832936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.832936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:50.832936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:50.832936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.832936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:50.832936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:50.832936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.832936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:50.833219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:50.833219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:50.833219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:50.833219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.833219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:50.833219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:50.833219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:50.833219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.833219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:50.833219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:50.833219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:50.833219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:50.833219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:50.833219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:50.833219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:50.833219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:50.833219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:50.833219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:50.833219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:50.833529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:50.833529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:50.833529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.833529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:50.850138 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:50.852467 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:50.852630 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:50.852753 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:50.853726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:50.853726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:50.854269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.856440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:50.856440 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:50.856440 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:50.856440 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:50.856440 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:50.856440 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:50.856440 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.856440 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:50.858935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.858935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:50.858935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:50.858935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:50.858935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.858935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:50.858935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:50.858935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.858935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:50.858935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:50.858935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:50.858935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.859226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:50.859226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:50.859226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:50.859226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:50.859226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:50.859226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:50.859226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:50.859226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:50.859226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:50.859226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:50.859226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:50.859226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:50.859226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:50.859726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.859726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:50.860199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.860199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:50.860199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:50.860199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:50.860199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:50.860199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:50.860199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.860199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:50.860199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:50.860199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.860199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:50.860199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:50.860199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.860199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:50.860199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:50.860199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:50.860573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:50.860573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:50.860573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.860573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:50.860573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:50.860573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:50.860573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:50.860573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:50.860573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:50.860573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:50.860573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:50.860573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:50.860573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:50.860573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:50.860573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:50.860573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:50.860573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:50.860573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:50.860573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:50.861039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:50.861039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:50.861039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:53.717855 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:53.718340 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:53.718413 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:53.719218 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:53.724890 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=239 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:33:53.758792 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:53.759644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:53.761179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:53.761179 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:53.761179 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:53.761179 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:53.761179 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:53.761179 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:53.761179 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:53.761179 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:53.762083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:53.762083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:53.762083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:53.762083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:53.762083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:53.762083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:53.762083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:53.762083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:53.762083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:53.762083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:53.762083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:53.762083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:53.762395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:53.762395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:53.762395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:53.762395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:53.762395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:53.762395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:53.762395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:53.762395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:53.762395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:53.762395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:53.762395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:53.762395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:53.762395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:53.762705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:53.762705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:53.763022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:53.763022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:53.763022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:53.763022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:53.763022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:53.763022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:53.763022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:53.763022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:53.763022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:53.763022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:53.763022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:53.763022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:53.763022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:53.763022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:53.763022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:53.763022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:53.763534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:53.763534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:53.763534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:53.763534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:53.763534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:53.763534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:53.763534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:53.763534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:53.763534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:53.763534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:53.763534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:53.763534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:53.763534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:53.763534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:53.763534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:53.763534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:53.763534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:53.763534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:53.763534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:53.764013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:53.764013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:53.764013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:55.790151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:55.790638 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:55.790676 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:55.791893 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:55.792080 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:55.792211 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:55.792345 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:55.792477 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:55.792578 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:55.792701 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:55.792813 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:55.792914 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:55.793021 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:55.793134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.793712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:55.793712 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:55.793712 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:55.793712 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:55.793712 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:55.793712 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:55.793712 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.793712 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:55.794821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.794821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:55.794821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:55.794821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:55.794821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.794821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:55.794821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:55.794821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.794821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:55.794821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:55.794821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:55.794821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.795069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:55.795069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:55.795069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:55.795069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:55.795069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:55.795069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:55.795069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:55.795069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:55.795069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:55.795069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:55.795069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:55.795069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:55.795069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:55.795277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.795277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:55.796524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.796524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:55.796524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:55.796524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:55.796524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:55.796524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:55.796524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.796524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:55.796524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:55.796524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.796524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:55.796524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:55.796524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.796524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:55.796524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:55.796524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.796807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:33:55.796807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:55.796807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.796807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:55.796807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:55.796807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:55.796807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.796807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:55.796807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:55.796807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:55.796807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:55.796807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:55.796807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:55.796807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:55.796807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:55.796807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:55.796807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:55.796807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:55.796807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:55.797105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:55.797105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.797105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:55.821914 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:55.822295 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:55.822418 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:55.822557 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:33:55.823045 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:55.823305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.823776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:55.823776 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:55.823776 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:55.823776 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:55.823776 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:55.823776 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:55.823776 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.823776 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:55.824540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.824540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:55.824540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:55.824540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:55.824540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.824540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:55.824540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:55.824540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:55.824540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.824540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:55.824540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:55.824540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:55.824540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:55.824540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:55.824540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:55.824967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:55.824967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:55.824967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:55.824967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:55.824967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:55.824967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:55.824967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:55.824967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.824967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:55.825618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.825618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:55.825618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:55.825618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:55.825618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.825618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:55.825618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:55.825618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:33:55.825618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:55.825618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:55.825618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.825618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:55.825943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:55.825943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:55.825943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.825943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:55.825943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:55.825943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:55.825943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:55.825943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:55.825943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:55.825943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:55.825943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:55.825943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:55.825943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:55.825943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:55.826299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:55.826299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:55.826299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.826299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:55.826696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.826696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:55.826696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:55.826696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:55.826696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:55.826696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:55.826696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.826696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:55.826696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:55.826696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.826696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:55.826696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:55.826696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.826696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:55.826696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:55.826696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.827582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:33:55.827582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:33:55.827582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:33:55.827582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:55.827582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:55.827582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.827582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:55.827582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:55.827582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:55.827582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.827582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:55.827582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:55.827582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:55.827582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:55.827582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:55.827582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:55.827582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:55.827582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:55.828341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:55.828341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:55.828341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:55.828341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:55.828341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:55.828341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.828341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:55.829810 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:55.830176 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:55.830317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:55.830436 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:55.830849 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:55.831099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.831649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:55.831649 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:55.831649 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:55.831649 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:55.831649 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:55.831649 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:55.831649 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.831649 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:55.832543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.832543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:33:55.832543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:55.832543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:55.832543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.832543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:55.832543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:55.832543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:33:55.832543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.832543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:33:55.832543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:55.832543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:33:55.832543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:55.832543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:33:55.832543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:33:55.832930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:55.832930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:33:55.832930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:55.832930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:33:55.832930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:33:55.832930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:55.832930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:33:55.832930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.832930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:33:55.833699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.833699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:55.833699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:55.833699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:55.833699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.833699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:55.833699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:55.833699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:55.833699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:55.833699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.833699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:55.834031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:55.834031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:55.834031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.834031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:55.834031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:55.834031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:55.834031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:55.834031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:55.834031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:55.834031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:55.834031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:55.834031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:55.834031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:55.834031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:55.834365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:55.834365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:55.834365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.834365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:55.834851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.834851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:55.834851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:55.834851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:55.834851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:55.834851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:55.834851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.834851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:55.834851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:55.834851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.834851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:55.834851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:55.834851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.834851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:55.834851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:55.834851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.834851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:33:55.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:33:55.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:33:55.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:33:55.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:55.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:55.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:55.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:55.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:55.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:55.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:55.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:55.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:55.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:55.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:55.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:55.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:55.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:55.835561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:55.835561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:55.835561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.835561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:55.851031 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:55.851366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:55.851526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:55.851725 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:55.852077 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:55.852372 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:55.852605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.853133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:55.853133 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:55.853133 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:55.853133 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:55.853133 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:55.853133 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:55.853133 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.853133 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:55.854037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.854037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:55.854037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:55.854037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:55.854037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.854037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:55.854037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:55.854037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.854037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:55.854037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:55.854037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:55.854037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.854344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:55.854344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:55.854344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:55.854344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:55.854344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:55.854344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:55.854344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:55.854344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:55.854344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:55.854344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:55.854344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:55.854344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:55.854344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:55.854531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.854531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:55.855103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.855103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:55.855103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:55.855103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:55.855103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:55.855103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:55.855103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.855103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:55.855103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:55.855103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.855103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:55.855103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:55.855103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.855103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:55.855103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:55.855103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:55.855413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:33:55.855413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:55.855413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.855413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:55.855413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:55.855413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:55.855413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:55.855413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:55.855413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:55.855413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:55.855413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:55.855413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:55.855413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:55.855413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:55.855413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:55.855413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:55.855413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:55.855413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:55.855413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:55.855696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:55.855696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:55.855696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:33:58.718915 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:33:58.719518 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:33:58.719518 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:33:58.720086 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:33:58.773817 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:33:58.774210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:58.775704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:33:58.775704 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:58.775704 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:33:58.775704 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:33:58.775704 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:58.775704 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:33:58.775704 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:58.775704 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:33:58.777136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:58.777136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:33:58.777136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:58.777136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:33:58.777136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:58.777136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:58.777136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:58.777136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:58.777136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:58.777136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:58.777136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:33:58.777136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:58.777504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:33:58.777504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:58.777504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:33:58.777504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:58.777504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:33:58.777504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:33:58.777504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:58.777504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:33:58.777504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:58.777504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:33:58.777504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:33:58.777504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:58.777504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:33:58.777827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:58.777827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:33:58.778573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:58.778573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:33:58.778573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:33:58.778573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:33:58.778573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:33:58.778573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:33:58.778573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:58.778573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:33:58.778573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:33:58.778573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:58.778573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:33:58.778573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:33:58.778573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:33:58.778573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:33:58.778573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:33:58.778573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:33:58.779025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:33:58.779025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:33:58.779025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:58.779025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:58.779025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:33:58.779025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:33:58.779025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:33:58.779025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:33:58.779025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:33:58.779025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:33:58.779025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:33:58.779025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:33:58.779025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:33:58.779025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:33:58.779025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:33:58.779025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:33:58.779025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:33:58.779025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:33:58.779025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:33:58.779811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:33:58.779811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:33:58.779811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:00.790603 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:00.792825 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:00.793541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:00.795736 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:00.795907 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:00.796093 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:00.796174 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:00.796300 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:00.796733 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:00.796733 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:00.796733 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:00.796733 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-e4e16d3b-c5d6-48a4-a4ab-c56e5984c74c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:00.796978 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:00.796978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.797372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:00.797372 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:00.797372 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:00.797372 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:00.797372 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:00.797372 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:00.797372 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.797372 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:00.800739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.800739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:00.800739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:00.800739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:00.800739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.800739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:00.800739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:00.800739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.800739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:00.800739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:00.800739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:00.800739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.801027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:00.801027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:00.801027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:00.801027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:00.801027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:00.801027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:00.801027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:00.801027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:00.801027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:00.801027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:00.801027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:00.801027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:00.801027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:00.801229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.801229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:00.801229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.801229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:00.801229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:00.801229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:00.801229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:00.801229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:00.801229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.801229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:00.801229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:00.801229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.801229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:00.801229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:00.801229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.801522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:00.801522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:00.801522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.801522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:00.801522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:00.801522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.801522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:00.801522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:00.801522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:00.801522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.801522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:00.801522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:00.801522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:00.801522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:00.801522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:00.801522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:00.801522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:00.801522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:00.801522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:00.801806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:00.801806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:00.801806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:00.801806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:00.801806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.801806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:00.822315 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:00.822843 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:00.822843 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:00.822843 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:34:00.823239 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:00.823239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.825515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:00.825515 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:00.825515 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:00.825515 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:00.825515 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:00.825515 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:00.825515 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.825515 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:00.825515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.825515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:00.825515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:00.825515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:00.825515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.825515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:00.825515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:00.825515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:00.825515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.825515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:00.826100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:00.826100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:00.826100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:00.826100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:00.826100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:00.826100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:00.826100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:00.826100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:00.826100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:00.826100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:00.826100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:00.826100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:00.826100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.826100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:00.826100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.826608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:00.826608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:00.826608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:00.826608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.826608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:00.826608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:00.826608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:34:00.826608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:00.826608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:00.826608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.826608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:00.826608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:00.826608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:00.826919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.826919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:00.826919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:00.826919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:00.826919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:00.826919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:00.826919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:00.826919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:00.826919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:00.826919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:00.826919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:00.826919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:00.826919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:00.826919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:00.827309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.827309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:00.827309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.827309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:00.827309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:00.827309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:00.827309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:00.827309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:00.827309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.827309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:00.827309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:00.827309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.827309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:00.827309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:00.827309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.827532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:00.827532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:00.827532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.827532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:00.827532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:00.827532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:34:00.827532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:00.827532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:00.827532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.827532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:00.827532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:00.827532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:00.827532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.827532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:00.827532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:00.827532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:00.827532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:00.827532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:00.827532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:00.827829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:00.827829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:00.827829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:00.827829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:00.827829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:00.827829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:00.827829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:00.827829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.827829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:00.830174 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:00.830732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:00.830732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:00.830732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:00.830975 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:00.831234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.833657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:00.833657 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:00.833657 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:00.833657 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:00.833657 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:00.833657 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:00.833657 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.833657 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:00.833657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.833657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:00.833657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:00.833657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:00.833657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.833657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:00.833657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:00.833657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:00.833657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.833657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:00.834447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:00.834447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:00.834447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:00.834447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:00.834447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:00.834447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:00.834447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:00.834447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:00.834447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:00.834447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:00.834447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:00.834447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:00.834447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.834447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:00.834447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.835117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:00.835117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:00.835117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:00.835117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.835117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:00.835117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:00.835117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:00.835117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:00.835117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.835117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:00.835117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:00.835117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:00.835117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.835404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:00.835404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:00.835404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:00.835404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:00.835404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:00.835404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:00.835404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:00.835404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:00.835404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:00.835404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:00.835404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:00.835404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:00.835404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:00.835633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.835633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:00.835633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.835633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:00.835633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:00.835633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:00.835633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:00.835633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:00.835633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.835633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:00.835633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:00.835633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.835633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:00.835633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:00.835633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.835873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:00.835873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:00.835873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.835873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:00.835873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:00.835873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:00.835873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:00.835873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.835873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:00.835873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:00.835873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:00.835873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.835873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:00.835873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:00.835873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:00.835873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:00.835873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:00.835873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:00.835873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:00.836194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:00.836194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:00.836194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:00.836194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:00.836194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:00.836194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:00.836194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.836194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:00.851386 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:00.851915 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:00.851915 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:00.851915 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:00.852135 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:00.852343 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:00.852562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.853006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:00.853006 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:00.853006 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:00.853006 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:00.853006 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:00.853006 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:00.853006 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.853006 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:00.854587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.854587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:00.854587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:00.854587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:00.854587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.854587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:00.854587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:00.854587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.854587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:00.854587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:00.854587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:00.854587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.854854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:00.854854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:00.854854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:00.854854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:00.854854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:00.854854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:00.854854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:00.854854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:00.854854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:00.854854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:00.854854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:00.854854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:00.854854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:00.855312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.855312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:00.855312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.855312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:00.855312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:00.855312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:00.855312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:00.855312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:00.855312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.855312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:00.855312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:00.855312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.855312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:00.855312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:00.855312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.855567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:00.855567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:00.855567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:00.855567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:00.855567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:00.855567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.855567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:00.855567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:00.855567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:00.855567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:00.855567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:00.855567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:00.855567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:00.855567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:00.855567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:00.855567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:00.855567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:00.855567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:00.855567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:00.855853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:00.855853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:00.855853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:00.855853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:00.855853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:00.855853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:03.721329 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:03.723313 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:03.723313 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:03.723443 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:03.762816 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:03.763229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:03.765377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:03.765377 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:03.765377 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:03.765377 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:03.765377 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:03.765377 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:03.765377 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:03.765377 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:03.765377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:03.765377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:03.765377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:03.765377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:03.765377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:03.765377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:03.765377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:03.765745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:03.765745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:03.765745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:03.765745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:03.765745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:03.765745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:03.765745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:03.765745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:03.765745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:03.765745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:03.765745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:03.765745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:03.765745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:03.765745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:03.766073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:03.766073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:03.766073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:03.766073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:03.766073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:03.766073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:03.766073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:03.766073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:03.766073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:03.766073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:03.766073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:03.766073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:03.766073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:03.766073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:03.766394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:03.766783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:03.766783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:03.766783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:03.766783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:03.766783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:03.766783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:03.766783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:03.766783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:03.766783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:05.793410 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:05.793873 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:05.794173 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:05.795457 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:05.795696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:05.795867 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:05.796108 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:05.796286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:05.797562 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:05.797737 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:05.797895 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:05.798056 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:05.798200 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f424fd1-dda0-4816-aa5c-f5e45222260b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:05.798370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.799820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:05.799820 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:05.799820 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:05.799820 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:05.799820 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:05.799820 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:05.799820 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.799820 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:05.801032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.801032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:05.801032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:05.801032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:05.801032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.801032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:05.801032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:05.801032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.801032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:05.801032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:05.801032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:05.801032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.801280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:05.801280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:05.801280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:05.801280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:05.801280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:05.801280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:05.801280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:05.801280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:05.801280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:05.801280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:05.801280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:05.801280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:05.801280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:05.801481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.801481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:05.801763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.801763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:05.801763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:05.801763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:05.801763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:05.801763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:05.801763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.801763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:05.801763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:05.801763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.801763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:05.801763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:05.801763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.801763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:05.801763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:05.801763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.802063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:05.802063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:05.802063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.802063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:05.802063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:05.802063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:05.802063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.802063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:05.802063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:05.802063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:05.802063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:05.802063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:05.802063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:05.802063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:05.802063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:05.802063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:05.802063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:05.802063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:05.802063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:05.802383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:05.802383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.802383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:05.822755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:05.824713 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:05.824829 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:05.824967 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:34:05.825356 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:05.825562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.826101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:05.826101 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:05.826101 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:05.826101 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:05.826101 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:05.826101 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:05.826101 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.826101 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:05.826872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.826872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:05.826872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:05.826872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:05.826872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.826872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:05.826872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:05.826872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:05.826872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.826872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:05.826872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:05.826872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:05.826872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:05.826872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:05.826872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:05.827318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:05.827318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:05.827318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:05.827318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:05.827318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:05.827318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:05.827318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:05.827318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.827318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:05.827854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.827854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:05.827854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:05.827854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:05.827854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.827854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:05.827854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:05.827854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:34:05.827854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:05.827854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:05.827854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.827854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:05.828115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:05.828115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:05.828115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.828115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:05.828115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:05.828115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:05.828115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:05.828115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:05.828115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:05.828115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:05.828115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:05.828115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:05.828115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:05.828115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:05.828493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:05.828493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:05.828493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.828493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:05.828938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.828938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:05.828938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:05.828938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:05.828938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:05.828938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:05.828938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.828938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:05.828938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:05.828938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.828938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:05.828938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:05.828938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.828938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:05.828938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:05.828938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.829235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:05.829235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:05.829235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:34:05.829235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:05.829235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:05.829235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.829235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:05.829235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:05.829235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:05.829235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.829235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:05.829235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:05.829235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:05.829235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:05.829235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:05.829235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:05.829235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:05.829235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:05.829538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:05.829538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:05.829538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:05.829538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:05.829538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:05.829538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.829538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:05.830600 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:05.831708 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:05.831827 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:05.831983 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:05.832699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:05.832960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.833501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:05.833501 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:05.833501 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:05.833501 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:05.833501 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:05.833501 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:05.833501 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.833501 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:05.834609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.834609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:05.834609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:05.834609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:05.834609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.834609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:05.834609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:05.834609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:05.834609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.834609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:05.834609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:05.834609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:05.834609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:05.834609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:05.834609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:05.834884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:05.834884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:05.834884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:05.834884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:05.834884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:05.834884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:05.834884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:05.834884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.834884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:05.835867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.835867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:05.835867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:05.835867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:05.835867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.835867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:05.835867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:05.835867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:05.835867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:05.835867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.835867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:05.836083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:05.836083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:05.836083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.836083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:05.836083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:05.836083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:05.836083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:05.836083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:05.836083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:05.836083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:05.836083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:05.836083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:05.836083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:05.836083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:05.836315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:05.836315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:05.836315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.836315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:05.837635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.837635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:05.837635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:05.837635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:05.837635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:05.837635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:05.837635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.837635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:05.837635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:05.837635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.837635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:05.837635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:05.837635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.837635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:05.837635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:05.837635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.837635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:05.837921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:05.837921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:05.837921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:05.837921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.837921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:05.837921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:05.837921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:05.837921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.837921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:05.837921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:05.837921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:05.837921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:05.837921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:05.837921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:05.837921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:05.837921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:05.837921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:05.837921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:05.837921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:05.838221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:05.838221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:05.838221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.838221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:05.851793 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:05.853037 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:05.853164 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:05.853371 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:05.855268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:05.855534 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:05.855867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.856428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:05.856428 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:05.856428 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:05.856428 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:05.856428 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:05.856428 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:05.856428 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.856428 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:05.858263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.858263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:05.858263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:05.858263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:05.858263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.858263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:05.858263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:05.858263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.858263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:05.858263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:05.858263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:05.858263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.858511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:05.858511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:05.858511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:05.858511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:05.858511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:05.858511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:05.858511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:05.858511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:05.858511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:05.858511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:05.858511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:05.858511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:05.858511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:05.858732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.858732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:05.861988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.861988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:05.861988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:05.861988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:05.861988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:05.861988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:05.861988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.861988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:05.861988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:05.861988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.861988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:05.861988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:05.861988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.861988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:05.861988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:05.861988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:05.862357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:05.862357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:05.862357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.862357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:05.862357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:05.862357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:05.862357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:05.862357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:05.862357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:05.862357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:05.862357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:05.862357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:05.862357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:05.862357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:05.862357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:05.862357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:05.862357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:05.862357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:05.862357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:05.862673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:05.862673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:05.862673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:08.722390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:08.722769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:08.722917 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:08.723599 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:08.761606 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:08.761778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:08.762337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:08.762337 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:08.762337 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:08.762337 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:08.762337 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:08.762337 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:08.762337 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:08.762337 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:08.763003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:08.763003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:08.763003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:08.763003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:08.763003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:08.763003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:08.763003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:08.763003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:08.763003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:08.763003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:08.763003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:08.763003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:08.763278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:08.763278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:08.763278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:08.763278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:08.763278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:08.763278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:08.763278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:08.763278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:08.763278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:08.763278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:08.763278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:08.763278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:08.763278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:08.763840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:08.763840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:08.763840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:08.763840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:08.763840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:08.763840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:08.763840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:08.763840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:08.763840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:08.763840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:08.763840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:08.763840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:08.763840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:08.763840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:08.763840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:08.764103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:08.764103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:08.764103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:08.764103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:08.764103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:08.764103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:08.764103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:08.764103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:08.764103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:08.764103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:08.764103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:08.764103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:08.764103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:08.764103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:08.764103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:08.764103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:08.764103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:08.764103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:08.764103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:08.764416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:08.764416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:08.764416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:08.764416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:08.764416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:08.764416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:10.794896 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:10.796964 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:10.796964 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:10.796964 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:10.796964 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:10.796964 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:10.797326 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:10.797326 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:10.797326 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:10.797469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:10.797886 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:10.797886 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:10.797886 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f424fd1-dda0-4816-aa5c-f5e45222260b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:10.797886 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e367330-4323-4718-94c3-86375e03fbd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:10.798064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.799883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:10.799883 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:10.799883 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:10.799883 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:10.799883 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:10.799883 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:10.799883 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.799883 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:10.804443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.804443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:10.804443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:10.804443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:10.804443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.804443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:10.804443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:10.804443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.804443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:10.804443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:10.804443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:10.804443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.804696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:10.804696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:10.804696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:10.804696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:10.804696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:10.804696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:10.804696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:10.804696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:10.804696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:10.804696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:10.804696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:10.804696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:10.804696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:10.805016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.805016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:10.807305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.807305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:10.807305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:10.807305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:10.807305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:10.807305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:10.807305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.807305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:10.807305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:10.807305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.807305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:10.807305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:10.807305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.807305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:10.807305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:10.807305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.807667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:10.807667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:10.807667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.807667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:10.807667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:10.807667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:10.807667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.807667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:10.807667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:10.807667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:10.807667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:10.807667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:10.807667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:10.807667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:10.807667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:10.807667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:10.807667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:10.807667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:10.807667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:10.808005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:10.808005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.808005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:10.824705 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:10.825077 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:10.825276 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:10.825434 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:34:10.825927 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:10.827545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.830268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:10.830268 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:10.830268 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:10.830268 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:10.830268 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:10.830268 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:10.830268 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.830268 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:10.830268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.830268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:10.830268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:10.830268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:10.830268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.830268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:10.830268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:10.830268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:10.830268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.830268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:10.830644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:10.830644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:10.830644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:10.830644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:10.830644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:10.830644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:10.830644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:10.830644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:10.830644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:10.830644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:10.830644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:10.830644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:10.830644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.830644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:10.830644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.830873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:10.830873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:10.830873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:10.830873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.830873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:10.830873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:10.830873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:34:10.830873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:10.830873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:10.830873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.830873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:10.830873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:10.830873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:10.831095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.831095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:10.831095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:10.831095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:10.831095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:10.831095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:10.831095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:10.831095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:10.831095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:10.831095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:10.831095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:10.831095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:10.831095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:10.831095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:10.831335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.831335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:10.831335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.831335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:10.831335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:10.831335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:10.831335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:10.831335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:10.831335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.831335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:10.831335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:10.831335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.831335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:10.831335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:10.831335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.831573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:10.831573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:10.831573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.831573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:10.831573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:10.831573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:34:10.831573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:10.831573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:10.831573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.831573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:10.831573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:10.831573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:10.831573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.831573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:10.831573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:10.831573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:10.831573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:10.831573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:10.831573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:10.831879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:10.831879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:10.831879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:10.831879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:10.831879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:10.831879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:10.831879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:10.831879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.831879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:10.835755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:10.835755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:10.835755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:10.835755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:10.835755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:10.835755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.835966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:10.835966 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:10.835966 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:10.835966 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:10.835966 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:10.835966 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:10.835966 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.835966 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:10.835966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.835966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:10.835966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:10.835966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:10.835966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.835966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:10.835966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:10.835966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:10.835966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.835966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:10.836699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:10.836699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:10.836699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:10.836699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:10.836699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:10.836699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:10.836699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:10.836699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:10.836699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:10.836699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:10.836699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:10.836699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:10.836699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.836699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:10.836699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.836979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:10.836979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:10.836979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:10.836979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.836979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:10.836979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:10.836979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:10.836979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:10.836979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.836979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:10.836979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:10.836979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:10.836979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.837169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:10.837169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:10.837169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:10.837169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:10.837169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:10.837169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:10.837169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:10.837169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:10.837169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:10.837169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:10.837169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:10.837169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:10.837169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:10.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:10.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:10.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:10.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:10.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:10.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:10.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:10.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:10.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:10.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:10.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.837612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:10.837612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:10.837612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.837612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:10.837612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:10.837612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:10.837612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:10.837612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.837612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:10.837612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:10.837612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:10.837612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.837612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:10.837612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:10.837612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:10.837612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:10.837612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:10.837612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:10.837612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:10.837917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:10.837917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:10.837917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:10.837917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:10.837917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:10.837917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:10.837917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.837917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:10.853081 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:10.853462 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:10.855466 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:10.855466 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:10.855466 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:10.855466 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:10.855466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.856085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:10.856085 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:10.856085 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:10.856085 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:10.856085 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:10.856085 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:10.856085 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.856085 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:10.856784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.856784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:10.856784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:10.856784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:10.856784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.856784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:10.856784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:10.856784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.856784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:10.856784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:10.856784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:10.856784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.857197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:10.857197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:10.857197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:10.857197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:10.857197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:10.857197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:10.857197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:10.857197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:10.857197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:10.857197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:10.857197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:10.857197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:10.857197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:10.857620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.857620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:10.857620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.857620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:10.857620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:10.857620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:10.857620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:10.857620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:10.857620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.857620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:10.857620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:10.857620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.857620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:10.857620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:10.857620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.858066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:10.858066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:10.858066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:10.858066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:10.858066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:10.858066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.858066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:10.858066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:10.858066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:10.858066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:10.858066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:10.858066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:10.858066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:10.858066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:10.858066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:10.858066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:10.858066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:10.858066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:10.858066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:10.858714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:10.858714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:10.858714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:10.858714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:10.858714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:10.858714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:13.722815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:13.723178 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:13.723178 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:13.723750 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:13.757719 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:13.757927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:13.758425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:13.758425 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:13.758425 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:13.758425 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:13.758425 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:13.758425 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:13.758425 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:13.758425 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:13.759125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:13.759125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:13.759125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:13.759125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:13.759125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:13.759125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:13.759125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:13.759125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:13.759125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:13.759125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:13.759125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:13.759125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:13.759426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:13.759426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:13.759426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:13.759426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:13.759426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:13.759426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:13.759426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:13.759426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:13.759426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:13.759426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:13.759426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:13.759426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:13.759426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:13.759650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:13.759650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:13.760073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:13.760073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:13.760073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:13.760073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:13.760073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:13.760073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:13.760073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:13.760073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:13.760073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:13.760073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:13.760073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:13.760073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:13.760073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:13.760073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:13.760073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:13.760073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:13.760494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:13.760494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:13.760494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:13.760494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:13.760494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:13.760494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:13.760494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:13.760494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:13.760494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:13.760494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:13.760494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:13.760494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:13.760494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:13.760494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:13.760494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:13.760494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:13.760494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:13.760494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:13.760494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:13.760863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:13.760863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:13.760863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:15.795389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:15.795903 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:15.795903 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:15.798688 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:15.798688 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:15.798688 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:15.798688 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:15.798688 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:15.798821 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:15.798821 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:15.798821 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:15.798821 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f424fd1-dda0-4816-aa5c-f5e45222260b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:15.798821 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:15.798949 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e367330-4323-4718-94c3-86375e03fbd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:15.798949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.798949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:15.798949 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:15.798949 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:15.798949 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:15.798949 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:15.798949 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:15.798949 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.798949 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:15.799520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.799520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:15.799520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:15.799520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:15.799520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.799520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:15.799520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:15.799520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.799520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:15.799520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:15.799520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:15.799520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.799734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:15.799734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:15.799734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:15.799734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:15.799734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:15.799734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:15.799734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:15.799734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:15.799734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:15.799734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:15.799734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:15.799734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:15.799734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:15.799966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.799966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:15.800294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.800294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:15.800294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:15.800294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:15.800294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:15.800294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:15.800294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.800294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:15.800294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:15.800294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.800294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:15.800294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:15.800294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.800294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:15.800294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:15.800294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.800556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:15.800556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:15.800556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.800556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:15.800556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:15.800556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:15.800556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.800556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:15.800556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:15.800556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:15.800556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:15.800556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:15.800556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:15.800556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:15.800556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:15.800556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:15.800556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:15.800556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:15.800556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:15.800842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:15.800842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.800842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:15.825120 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:15.825422 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:15.825552 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:15.825691 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:34:15.826690 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:15.826690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.826883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:15.826883 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:15.826883 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:15.826883 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:15.826883 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:15.826883 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:15.826883 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.826883 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:15.827805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.827805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:15.827805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:15.827805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:15.827805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.827805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:15.827805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:15.827805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:15.827805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.827805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:15.827805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:15.827805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:15.827805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:15.827805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:15.827805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:15.828076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:15.828076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:15.828076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:15.828076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:15.828076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:15.828076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:15.828076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:15.828076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.828076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:15.828763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.828763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:15.828763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:15.828763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:15.828763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.828763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:15.828763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:15.828763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:34:15.828763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:15.828763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:15.828763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.828763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:15.828985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:15.828985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:15.828985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.828985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:15.828985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:15.828985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:15.828985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:15.828985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:15.828985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:15.828985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:15.828985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:15.828985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:15.828985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:15.828985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:15.829198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:15.829198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:15.829198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.829198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:15.829904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.829904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:15.829904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:15.829904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:15.829904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:15.829904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:15.829904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.829904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:15.829904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:15.829904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.829904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:15.829904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:15.829904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.829904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:15.829904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:15.829904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.830315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:15.830315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:15.830315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:34:15.830315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:15.830315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:15.830315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.830315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:15.830315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:15.830315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:15.830315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.830315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:15.830315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:15.830315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:15.830315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:15.830315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:15.830315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:15.830315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:15.830315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:15.830637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:15.830637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:15.830637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:15.830637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:15.830637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:15.830637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.830637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:15.833239 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:15.833522 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:15.833641 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:15.833766 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:15.834515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:15.834515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.834652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:15.834652 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:15.834652 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:15.834652 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:15.834652 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:15.834652 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:15.834652 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.834652 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:15.835467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.835467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:15.835467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:15.835467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:15.835467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.835467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:15.835467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:15.835467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:15.835467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.835467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:15.835467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:15.835467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:15.835467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:15.835467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:15.835467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:15.835699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:15.835699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:15.835699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:15.835699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:15.835699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:15.835699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:15.835699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:15.835699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.835699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:15.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:15.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:15.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:15.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:15.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:15.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:15.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:15.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:15.836538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:15.836538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:15.836538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.836538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:15.836538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:15.836538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:15.836538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:15.836538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:15.836538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:15.836538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:15.836538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:15.836538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:15.836538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:15.836538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:15.836749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:15.836749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:15.836749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.836749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:15.837481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.837481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:15.837481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:15.837481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:15.837481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:15.837481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:15.837481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.837481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:15.837481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:15.837481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.837481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:15.837481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:15.837481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.837481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:15.837481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:15.837481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.837481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:15.837779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:15.837779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:15.837779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:15.837779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.837779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:15.837779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:15.837779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:15.837779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.837779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:15.837779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:15.837779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:15.837779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:15.837779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:15.837779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:15.837779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:15.837779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:15.837779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:15.837779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:15.837779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:15.838064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:15.838064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:15.838064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.838064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:15.853495 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:15.853831 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:15.853977 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:15.854179 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:15.855763 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:15.855763 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:15.855763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.856323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:15.856323 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:15.856323 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:15.856323 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:15.856323 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:15.856323 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:15.856323 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.856323 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:15.856984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.856984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:15.856984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:15.856984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:15.856984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.856984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:15.856984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:15.856984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.856984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:15.856984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:15.856984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:15.856984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.857363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:15.857363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:15.857363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:15.857363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:15.857363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:15.857363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:15.857363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:15.857363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:15.857363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:15.857363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:15.857363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:15.857363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:15.857363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:15.857816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.857816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:15.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:15.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:15.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:15.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:15.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:15.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:15.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:15.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:15.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:15.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:15.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:15.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:15.858512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:15.858512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:15.858512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.858512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:15.858512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:15.858512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:15.858512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:15.858512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:15.858512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:15.858512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:15.858512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:15.858512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:15.858512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:15.858512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:15.858512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:15.858512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:15.858512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:15.858512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:15.858512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:15.859061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:15.859061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:15.859061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:18.723416 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:18.723851 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:18.724305 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:18.724975 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:18.764898 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:18.765076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:18.765512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:18.765512 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:18.765512 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:18.765512 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:18.765512 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:18.765512 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:18.765512 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:18.765512 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:18.766156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:18.766156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:18.766156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:18.766156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:18.766156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:18.766156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:18.766156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:18.766156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:18.766156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:18.766156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:18.766156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:18.766156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:18.766581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:18.766581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:18.766581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:18.766581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:18.766581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:18.766581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:18.766581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:18.766581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:18.766581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:18.766581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:18.766581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:18.766581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:18.766581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:18.766968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:18.766968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:18.766968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:18.766968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:18.766968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:18.766968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:18.766968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:18.766968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:18.766968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:18.766968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:18.766968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:18.766968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:18.766968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:18.766968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:18.766968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:18.767361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:18.767361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:18.767361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:18.767361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:18.767361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:18.767361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:18.767361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:18.767361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:18.767361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:18.767361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:18.767361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:18.767361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:18.767361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:18.767361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:18.767361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:18.767361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:18.767361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:18.767361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:18.767361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:18.767832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:18.767832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:18.767832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:18.767832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:18.767832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:18.767832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:20.795758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:20.796139 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:20.796139 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:20.797183 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:20.797408 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:20.797563 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:20.797704 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:20.797844 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:20.798003 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:20.798135 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:20.798353 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:20.798453 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f424fd1-dda0-4816-aa5c-f5e45222260b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:20.798638 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:20.798788 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e367330-4323-4718-94c3-86375e03fbd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:20.798954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.799404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:20.799404 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:20.799404 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:20.799404 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:20.799404 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:20.799404 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:20.799404 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.799404 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:20.800140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.800140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:20.800140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:20.800140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:20.800140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.800140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:20.800140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:20.800140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.800140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:20.800140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:20.800140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:20.800140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.800409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:20.800409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:20.800409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:20.800409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:20.800409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:20.800409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:20.800409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:20.800409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:20.800409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:20.800409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:20.800409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:20.800409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:20.800409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:20.800651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.800651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:20.801417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.801417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:20.801417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:20.801417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:20.801417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:20.801417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:20.801417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.801417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:20.801417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:20.801417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.801417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:20.801417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:20.801417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.801417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:20.801417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:20.801417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.801813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:20.801813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:20.801813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.801813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:20.801813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:20.801813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:20.801813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.801813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:20.801813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:20.801813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:20.801813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:20.801813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:20.801813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:20.801813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:20.801813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:20.801813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:20.801813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:20.801813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:20.801813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:20.802149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:20.802149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.802149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:20.825458 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:20.825691 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:20.825777 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:20.825868 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:34:20.826191 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:20.826425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.826824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:20.826824 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:20.826824 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:20.826824 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:20.826824 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:20.826824 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:20.826824 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.826824 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:20.827494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.827494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:20.827494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:20.827494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:20.827494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.827494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:20.827494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:20.827494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:20.827494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.827494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:20.827494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:20.827494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:20.827494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:20.827494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:20.827494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:20.827756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:20.827756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:20.827756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:20.827756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:20.827756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:20.827756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:20.827756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:20.827756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.827756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:20.828378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.828378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:20.828378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:20.828378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:20.828378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.828378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:20.828378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:20.828378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:34:20.828378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:20.828378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:20.828378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.828378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:20.828639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:20.828639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:20.828639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.828639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:20.828639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:20.828639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:20.828639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:20.828639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:20.828639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:20.828639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:20.828639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:20.828639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:20.828639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:20.828639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:20.828910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:20.828910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:20.828910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.828910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:20.829787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.829787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:20.829787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:20.829787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:20.829787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:20.829787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:20.829787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.829787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:20.829787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:20.829787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.829787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:20.829787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:20.829787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.829787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:20.829787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:20.829787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.830119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:20.830119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:20.830119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:34:20.830119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:20.830119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:20.830119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.830119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:20.830119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:20.830119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:20.830119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.830119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:20.830119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:20.830119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:20.830119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:20.830119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:20.830119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:20.830119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:20.830119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:20.830467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:20.830467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:20.830467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:20.830467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:20.830467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:20.830467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.830467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:20.833561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:20.833743 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:20.833827 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:20.833924 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:20.834188 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:20.834467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.834804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:20.834804 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:20.834804 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:20.834804 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:20.834804 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:20.834804 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:20.834804 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.834804 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:20.835448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.835448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:20.835448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:20.835448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:20.835448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.835448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:20.835448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:20.835448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:20.835448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.835448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:20.835448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:20.835448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:20.835448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:20.835448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:20.835448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:20.835694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:20.835694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:20.835694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:20.835694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:20.835694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:20.835694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:20.835694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:20.835694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.835694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:20.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:20.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:20.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:20.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:20.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:20.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:20.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:20.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.836347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:20.836541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:20.836541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:20.836541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.836541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:20.836541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:20.836541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:20.836541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:20.836541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:20.836541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:20.836541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:20.836541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:20.836541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:20.836541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:20.836541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:20.836776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:20.836776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:20.836776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.836776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:20.837520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.837520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:20.837520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:20.837520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:20.837520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:20.837520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:20.837520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.837520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:20.837520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:20.837520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.837520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:20.837520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:20.837520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.837520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:20.837520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:20.837520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.837520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:20.837820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:20.837820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:20.837820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:20.837820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.837820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:20.837820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:20.837820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:20.837820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.837820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:20.837820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:20.837820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:20.837820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:20.837820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:20.837820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:20.837820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:20.837820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:20.837820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:20.837820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:20.837820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:20.838139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:20.838139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:20.838139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.838139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:20.854352 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:20.854591 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:20.854660 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:20.854806 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:20.855156 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:20.855460 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:20.855753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.856217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:20.856217 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:20.856217 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:20.856217 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:20.856217 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:20.856217 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:20.856217 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.856217 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:20.856976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.856976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:20.856976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:20.856976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:20.856976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.856976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:20.856976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:20.856976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.856976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:20.856976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:20.856976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:20.856976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.857237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:20.857237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:20.857237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:20.857237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:20.857237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:20.857237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:20.857237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:20.857237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:20.857237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:20.857237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:20.857237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:20.857237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:20.857237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:20.857492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.857492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:20.858053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.858053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:20.858053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:20.858053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:20.858053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:20.858053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:20.858053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.858053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:20.858053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:20.858053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.858053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:20.858053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:20.858053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.858053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:20.858053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:20.858053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:20.858682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:20.858682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:20.858682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:23.723917 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:23.724469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:23.724469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:23.725025 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:23.756893 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:23.757066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:23.757628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:23.757628 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:23.757628 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:23.757628 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:23.757628 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:23.757628 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:23.757628 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:23.757628 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:23.758380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:23.758380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:23.758380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:23.758380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:23.758380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:23.758380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:23.758380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:23.758380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:23.758380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:23.758380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:23.758380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:23.758380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:23.758669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:23.758669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:23.758669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:23.758669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:23.758669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:23.758669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:23.758669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:23.758669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:23.758669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:23.758669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:23.758669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:23.758669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:23.758669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:23.758895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:23.758895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:23.759413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:23.759413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:23.759413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:23.759413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:23.759413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:23.759413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:23.759413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:23.759413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:23.759413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:23.759413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:23.759413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:23.759413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:23.759413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:23.759413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:23.759413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:23.759413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:23.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:23.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:23.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:23.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:23.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:23.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:23.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:23.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:23.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:23.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:23.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:23.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:23.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:23.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:23.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:23.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:23.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:23.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:23.759713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:23.760107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:23.760107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:23.760107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:25.796194 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:25.796704 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:25.796746 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:25.798260 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:25.798459 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:25.798651 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:25.798806 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:25.799778 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:25.799971 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:25.800266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:25.800434 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:25.800737 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f424fd1-dda0-4816-aa5c-f5e45222260b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:25.801412 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:25.802557 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e367330-4323-4718-94c3-86375e03fbd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:25.802754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.804533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:25.804533 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:25.804533 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:25.804533 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:25.804533 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:25.804533 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:25.804533 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.804533 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:25.810499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.810499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:25.810499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:25.810499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:25.810499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.810499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:25.810499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:25.810499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.810499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:25.810499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:25.810499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:25.810499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.810772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:25.810772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:25.810772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:25.810772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:25.810772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:25.810772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:25.810772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:25.810772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:25.810772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:25.810772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:25.810772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:25.810772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:25.810772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:25.811005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.811005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:25.811005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.811005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:25.811005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:25.811005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:25.811005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:25.811005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:25.811005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.811005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:25.811005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:25.811005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.811005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:25.811005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:25.811005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.811332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:25.811332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:25.811332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.811332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:25.811332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:25.811332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.811332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:25.811332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:25.811332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:25.811332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.811332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:25.811332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:25.811332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:25.811332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:25.811332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:25.811332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:25.811332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:25.811332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:25.811332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:25.811649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:25.811649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:25.811649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:25.811649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:25.811649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.811649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:25.825841 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:25.826202 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:25.826311 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:25.826409 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:34:25.827264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:25.827448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.828049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:25.828049 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:25.828049 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:25.828049 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:25.828049 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:25.828049 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:25.828049 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.828049 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:25.828696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.828696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:25.828696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:25.828696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:25.828696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.828696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:25.828696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:25.828696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:25.828696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.828696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:25.828696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:25.828696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:25.828696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:25.828696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:25.828696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:25.829163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:25.829163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:25.829163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:25.829163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:25.829163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:25.829163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:25.829163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:25.829163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.829163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:25.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:25.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:25.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:25.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:25.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:25.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:34:25.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:25.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:25.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:25.829778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:25.829778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:25.829778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.829778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:25.829778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:25.829778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:25.829778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:25.829778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:25.829778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:25.829778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:25.829778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:25.829778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:25.829778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:25.829778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:25.830155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:25.830155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:25.830155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.830155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:25.830155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.830155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:25.830155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:25.830155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:25.830155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:25.830155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:25.830155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.830155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:25.830155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:25.830155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.831613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:25.831613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:25.831613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.831613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:25.831613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:25.831613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.831613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:25.831613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:25.831613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:34:25.831613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:25.831613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:25.831613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.831613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:25.831613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:25.831613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:25.831613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.831613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:25.831613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:25.831613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:25.832146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:25.832146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:25.832146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:25.832146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:25.832146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:25.832146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:25.832146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:25.832146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:25.832146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:25.832146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:25.832146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.832146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:25.833847 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:25.835089 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:25.835089 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:25.835089 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:25.838397 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:25.838589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.839112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:25.839112 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:25.839112 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:25.839112 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:25.839112 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:25.839112 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:25.839112 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.839112 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:25.839827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.839827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:25.839827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:25.839827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:25.839827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.839827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:25.839827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:25.839827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:25.839827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.839827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:25.839827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:25.839827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:25.839827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:25.839827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:25.839827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:25.840220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:25.840220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:25.840220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:25.840220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:25.840220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:25.840220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:25.840220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:25.840220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.840220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:25.840530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.840530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:25.840530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:25.840530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:25.840530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.840530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:25.840530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:25.840530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:25.840530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:25.840530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.840530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:25.840828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:25.840828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:25.840828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.840828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:25.840828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:25.840828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:25.840828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:25.840828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:25.840828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:25.840828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:25.840828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:25.840828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:25.840828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:25.840828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:25.841510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:25.841510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:25.841510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.841510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:25.841510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.841510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:25.841510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:25.841510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:25.841510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:25.841510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:25.841510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.841510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:25.841510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:25.841510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.841876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:25.841876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:25.841876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.841876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:25.841876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:25.841876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.841876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:25.841876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:25.841876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:25.841876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:25.841876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.841876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:25.841876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:25.841876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:25.841876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.841876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:25.841876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:25.841876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:25.841876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:25.843537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:25.843537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:25.843537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:25.843537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:25.843537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:25.843537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:25.843537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:25.843537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:25.843537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:25.843537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.843537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:25.854902 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:25.855352 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:25.855637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:25.855637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:25.855945 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:25.856139 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:25.856415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.857961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:25.857961 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:25.857961 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:25.857961 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:25.857961 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:25.857961 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:25.857961 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.857961 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:25.858695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.858695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:25.858695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:25.858695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:25.858695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.858695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:25.858695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:25.858695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.858695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:25.858695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:25.858695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:25.858695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.859036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:25.859036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:25.859036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:25.859036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:25.859036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:25.859036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:25.859036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:25.859036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:25.859036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:25.859036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:25.859036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:25.859036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:25.859036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:25.859449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.859449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:25.862310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.862310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:25.862310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:25.862310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:25.862310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:25.862310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:25.862310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.862310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:25.862310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:25.862310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.862310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:25.862310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:25.862310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.862310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:25.862310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:25.862310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:25.862890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:25.862890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:25.862890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.862890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:25.862890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:25.862890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:25.862890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:25.862890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:25.862890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:25.862890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:25.862890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:25.862890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:25.862890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:25.862890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:25.862890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:25.862890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:25.862890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:25.862890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:25.862890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:25.863485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:25.863485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:25.863485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:28.724376 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:28.724818 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:28.724924 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:28.725689 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:28.761737 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:28.761864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:28.762357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:28.762357 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:28.762357 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:28.762357 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:28.762357 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:28.762357 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:28.762357 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:28.762357 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:28.763217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:28.763217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:28.763217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:28.763217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:28.763217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:28.763217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:28.763217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:28.763217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:28.763217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:28.763217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:28.763217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:28.763217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:28.763529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:28.763529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:28.763529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:28.763529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:28.763529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:28.763529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:28.763529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:28.763529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:28.763529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:28.763529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:28.763529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:28.763529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:28.763529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:28.765191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:28.765191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:28.765191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:28.765191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:28.765191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:28.765191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:28.765191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:28.765191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:28.765191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:28.765191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:28.765191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:28.765191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:28.765191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:28.765191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:28.765191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:28.765599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:28.765599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:28.765599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:28.765599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:28.765599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:28.765599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:28.765599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:28.765599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:28.765599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:28.765599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:28.765599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:28.765599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:28.765599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:28.765599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:28.765599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:28.765599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:28.765599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:28.765599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:28.765599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:28.766001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:28.766001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:28.766001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:28.766001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:28.766001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:28.766001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:30.798221 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:30.798221 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:30.798221 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:30.798221 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:30.798918 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:30.798918 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:30.798918 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:30.798918 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:30.798918 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:30.799112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:30.799112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f424fd1-dda0-4816-aa5c-f5e45222260b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:30.799112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:30.799214 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e367330-4323-4718-94c3-86375e03fbd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:30.799392 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:30.800663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.800663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:30.800663 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:30.800663 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:30.800663 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:30.800663 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:30.800663 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:30.800663 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.800663 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:30.802190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.802190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:30.802190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:30.802190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:30.802190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.802190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:30.802190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:30.802190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.802190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:30.802190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:30.802190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:30.802190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.802915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:30.802915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:30.802915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:30.802915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:30.802915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:30.802915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:30.802915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:30.802915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:30.802915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:30.802915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:30.802915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:30.802915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:30.802915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:30.803361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.803361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:30.803361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.803361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:30.803361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:30.803361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:30.803361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:30.803361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:30.803361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.803361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:30.803361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:30.803361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.803361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:30.803361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:30.803361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.803695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:30.803695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:30.803695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.803695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:30.803695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:30.803695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.803695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:30.803695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:30.803695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:30.803695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.803695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:30.803695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:30.803695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:30.803695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:30.803695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:30.803695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:30.803695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:30.803695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:30.803695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:30.804065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:30.804065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:30.804065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:30.804065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:30.804065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.804065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:30.826223 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:30.826637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:30.826817 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:30.827075 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:34:30.829344 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:30.829611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.833224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:30.833224 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:30.833224 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:30.833224 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:30.833224 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:30.833224 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:30.833224 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.833224 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:30.834043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.834043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:30.834043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:30.834043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:30.834043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.834043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:30.834043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:30.834043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:30.834043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.834043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:30.834043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:30.834043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:30.834043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:30.834043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:30.834043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:30.834299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:30.834299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:30.834299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:30.834299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:30.834299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:30.834299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:30.834299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:30.834299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.834299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:30.835073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.835073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:30.835073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:30.835073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:30.835073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.835073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:30.835073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:30.835073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:34:30.835073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:30.835073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:30.835073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.835073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:30.835323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:30.835323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:30.835323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.835323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:30.835323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:30.835323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:30.835323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:30.835323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:30.835323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:30.835323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:30.835323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:30.835323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:30.835323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:30.835323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:30.835580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:30.835580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:30.835580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.835580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:30.836519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.836519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:30.836519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:30.836519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:30.836519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:30.836519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:30.836519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.836519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:30.836519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:30.836519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.836519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:30.836519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:30.836519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.836519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:30.836519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:30.836519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.836829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:30.836829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:30.836829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:34:30.836829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:30.836829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:30.836829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.836829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:30.836829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:30.836829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:30.836829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.836829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:30.836829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:30.836829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:30.836829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:30.836829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:30.836829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:30.836829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:30.836829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:30.837305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:30.837305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:30.837305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:30.837305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:30.837305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:30.837305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.837305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:30.837305 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:30.837802 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:30.837967 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:30.838085 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:30.838575 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:30.839523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.839523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:30.839523 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:30.839523 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:30.839523 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:30.839523 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:30.839523 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:30.839523 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.839523 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:30.839990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.839990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:30.839990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:30.839990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:30.839990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.839990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:30.839990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:30.839990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:30.839990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.839990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:30.839990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:30.839990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:30.839990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:30.839990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:30.839990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:30.840261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:30.840261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:30.840261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:30.840261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:30.840261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:30.840261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:30.840261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:30.840261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.840261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:30.840712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.840712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:30.840712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:30.840712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:30.840712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.840712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:30.840712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:30.840712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:30.840712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:30.840712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.840712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:30.840970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:30.840970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:30.840970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.840970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:30.840970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:30.840970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:30.840970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:30.840970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:30.840970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:30.840970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:30.840970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:30.840970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:30.840970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:30.840970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:30.841425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:30.841425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:30.841425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.841425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:30.841540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.841540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:30.841540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:30.841540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:30.841540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:30.841540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:30.841540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.841540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:30.841540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:30.841540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.841540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:30.841540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:30.841540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.841540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:30.841540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:30.841540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.841540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:30.841968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:30.841968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:30.841968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:30.841968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.841968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:30.841968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:30.841968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:30.841968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.841968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:30.841968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:30.841968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:30.841968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:30.841968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:30.841968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:30.841968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:30.841968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:30.841968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:30.841968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:30.841968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:30.842487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:30.842487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:30.842487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.842487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:30.855945 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:30.855945 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:30.856357 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:30.856357 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:30.856620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:30.857008 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:30.857603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.859720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:30.859720 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:30.859720 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:30.859720 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:30.859720 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:30.859720 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:30.859720 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.859720 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:30.860468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.860468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:30.860468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:30.860468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:30.860468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.860468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:30.860468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:30.860468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.860468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:30.860468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:30.860468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:30.860468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.860842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:30.860842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:30.860842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:30.860842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:30.860842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:30.860842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:30.860842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:30.860842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:30.860842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:30.860842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:30.860842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:30.860842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:30.860842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:30.861460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.861460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:30.861460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.861460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:30.861460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:30.861460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:30.861460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:30.861460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:30.861460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.861460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:30.861460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:30.861460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.861460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:30.861460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:30.861460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.861846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:30.861846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:30.861846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:30.861846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:30.861846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:30.861846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.861846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:30.861846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:30.861846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:30.861846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:30.861846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:30.861846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:30.861846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:30.861846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:30.861846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:30.861846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:30.861846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:30.861846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:30.861846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:30.862488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:30.862488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:30.862488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:30.862488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:30.862488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:30.862488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:33.724804 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:33.726769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:33.726769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:33.726769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:33.772727 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:33.772983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:33.774392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:33.774392 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:33.774392 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:33.774392 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:33.774392 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:33.774392 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:33.774392 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:33.774392 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:33.775627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:33.775627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:33.775627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:33.775627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:33.775627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:33.775627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:33.775627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:33.775627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:33.775627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:33.775627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:33.775627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:33.775627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:33.776323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:33.776323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:33.776323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:33.776323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:33.776323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:33.776323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:33.776323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:33.776323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:33.776323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:33.776323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:33.776323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:33.776323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:33.776323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:33.776830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:33.776830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:33.776830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:33.776830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:33.776830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:33.776830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:33.776830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:33.776830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:33.776830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:33.776830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:33.776830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:33.776830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:33.776830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:33.776830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:33.776830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:33.777480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:33.777480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:33.777480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:33.777480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:33.777480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:33.777480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:33.777480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:33.777480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:33.777480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:33.777480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:33.777480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:33.777480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:33.777480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:33.777480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:33.777480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:33.777480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:33.777480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:33.777480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:33.777480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:33.778209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:33.778209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:33.778209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:33.778209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:33.778209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:33.778209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:35.797115 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:35.797601 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:35.797643 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:35.798606 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:35.798754 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:35.798867 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:35.799094 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:35.799191 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:35.799331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:35.799436 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:35.799536 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e367330-4323-4718-94c3-86375e03fbd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:35.799631 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:35.799755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f424fd1-dda0-4816-aa5c-f5e45222260b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:35.799854 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:35.799964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.800385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:35.800385 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:35.800385 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:35.800385 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:35.800385 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:35.800385 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:35.800385 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.800385 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:35.801038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.801038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:35.801038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:35.801038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:35.801038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.801038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:35.801038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:35.801038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.801038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:35.801038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:35.801038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:35.801038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.801300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:35.801300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:35.801300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:35.801300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:35.801300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:35.801300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:35.801300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:35.801300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:35.801300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:35.801300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:35.801300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:35.801300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:35.801300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:35.801506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.801506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:35.801786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.801786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:35.801786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:35.801786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:35.801786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:35.801786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:35.801786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.801786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:35.801786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:35.801786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.801786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:35.801786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:35.801786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.801786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:35.801786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:35.801786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.802100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:35.802100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:35.802100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.802100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:35.802100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:35.802100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:35.802100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.802100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:35.802100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:35.802100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:35.802100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:35.802100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:35.802100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:35.802100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:35.802100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:35.802100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:35.802100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:35.802100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:35.802100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:35.802475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:35.802475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.802475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:35.826657 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:35.827020 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:35.827142 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:35.827268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:34:35.827587 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:35.827764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.828169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:35.828169 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:35.828169 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:35.828169 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:35.828169 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:35.828169 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:35.828169 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.828169 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:35.828735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.828735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:35.828735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:35.828735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:35.828735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.828735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:35.828735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:35.828735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:35.828735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.828735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:35.828735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:35.828735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:35.828735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:35.828735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:35.828735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:35.829048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:35.829048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:35.829048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:35.829048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:35.829048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:35.829048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:35.829048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:35.829048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.829048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:35.829351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.829351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:35.829351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:35.829351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:35.829351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.829351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:35.829351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:35.829351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:34:35.829351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:35.829351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:35.829351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.829351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:35.829596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:35.829596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:35.829596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.829596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:35.829596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:35.829596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:35.829596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:35.829596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:35.829596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:35.829596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:35.829596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:35.829596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:35.829596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:35.829596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:35.829831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:35.829831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:35.829831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.829831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:35.830107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.830107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:35.830107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:35.830107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:35.830107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:35.830107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:35.830107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.830107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:35.830107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:35.830107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.830107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:35.830107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:35.830107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.830107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:35.830107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:35.830107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.830426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:35.830426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:35.830426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:34:35.830426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:35.830426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:35.830426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.830426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:35.830426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:35.830426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:35.830426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.830426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:35.830426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:35.830426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:35.830426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:35.830426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:35.830426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:35.830426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:35.830426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:35.830727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:35.830727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:35.830727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:35.830727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:35.830727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:35.830727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.830727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:35.837952 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:35.838248 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:35.838363 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:35.838487 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:35.838838 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:35.839065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.839815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:35.839815 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:35.839815 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:35.839815 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:35.839815 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:35.839815 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:35.839815 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.839815 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:35.841006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.841006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:35.841006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:35.841006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:35.841006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.841006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:35.841006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:35.841006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:35.841006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.841006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:35.841006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:35.841006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:35.841006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:35.841006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:35.841006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:35.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:35.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:35.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:35.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:35.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:35.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:35.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:35.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:35.841699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.841699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:35.841699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:35.841699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:35.841699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.841699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:35.841699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:35.841699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:35.841699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:35.841699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.841699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:35.841936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:35.841936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:35.841936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.841936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:35.841936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:35.841936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:35.841936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:35.841936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:35.841936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:35.841936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:35.841936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:35.841936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:35.841936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:35.841936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:35.842461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:35.842461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:35.842461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.842461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:35.843265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.843265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:35.843265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:35.843265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:35.843265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:35.843265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:35.843265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.843265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:35.843265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:35.843265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.843265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:35.843265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:35.843265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.843265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:35.843265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:35.843265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.843265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:35.843606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:35.843606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:35.843606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:35.843606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.843606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:35.843606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:35.843606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:35.843606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.843606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:35.843606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:35.843606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:35.843606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:35.843606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:35.843606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:35.843606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:35.843606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:35.843606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:35.843606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:35.843606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:35.843904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:35.843904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:35.843904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.843904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:35.855971 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:35.856482 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:35.856580 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:35.856717 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:35.857232 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:35.857522 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:35.857859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.858457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:35.858457 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:35.858457 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:35.858457 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:35.858457 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:35.858457 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:35.858457 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.858457 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:35.859162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.859162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:35.859162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:35.859162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:35.859162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.859162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:35.859162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:35.859162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.859162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:35.859162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:35.859162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:35.859162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.859427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:35.859427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:35.859427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:35.859427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:35.859427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:35.859427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:35.859427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:35.859427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:35.859427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:35.859427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:35.859427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:35.859427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:35.859427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:35.859669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.859669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:35.859961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.859961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:35.859961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:35.859961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:35.859961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:35.859961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:35.859961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.859961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:35.859961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:35.859961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.859961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:35.859961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:35.859961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.859961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:35.859961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:35.859961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:35.860246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:35.860246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:35.860246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.860246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:35.860246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:35.860246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:35.860246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:35.860246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:35.860246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:35.860246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:35.860246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:35.860246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:35.860246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:35.860246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:35.860246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:35.860246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:35.860246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:35.860246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:35.860246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:35.860591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:35.860591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:35.860591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:38.725570 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:38.727587 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:38.727587 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:38.727587 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:38.761848 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:38.762047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:38.764049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:38.764049 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:38.764049 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:38.764049 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:38.764049 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:38.764049 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:38.764049 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:38.764049 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:38.764049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:38.764049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:38.764049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:38.764049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:38.764049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:38.764049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:38.764049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:38.764474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:38.764474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:38.764474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:38.764474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:38.764474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:38.764474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:38.764474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:38.764474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:38.764474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:38.764474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:38.764474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:38.764474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:38.764474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:38.764474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:38.764710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:38.764710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:38.764710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:38.764710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:38.764710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:38.764710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:38.764710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:38.764710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:38.764710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:38.764710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:38.764710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:38.764710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:38.764710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:38.764710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:38.765015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:38.765375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:38.765375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:38.765375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:38.765375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:38.765375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:38.765375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:38.765375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:38.765375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:38.765375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:40.797554 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:40.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:40.799143 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:40.799453 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:40.799638 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:40.799762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:40.799922 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:40.800044 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:40.800153 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:40.800273 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e367330-4323-4718-94c3-86375e03fbd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:40.800384 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:40.800483 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f424fd1-dda0-4816-aa5c-f5e45222260b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:40.800582 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:40.800679 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:40.800780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.802841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:40.802841 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:40.802841 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:40.802841 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:40.802841 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:40.802841 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:40.802841 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.802841 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:40.802841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.802841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:40.802841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:40.802841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:40.802841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.802841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:40.802841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:40.803363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.803363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:40.803363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:40.803363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:40.803363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.803363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:40.803363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:40.803363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:40.803363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:40.803363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:40.803363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:40.803363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:40.803363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:40.803363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:40.803774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:40.803774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:40.803774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:40.803774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:40.803774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.803774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:40.803774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.803774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:40.803774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:40.803774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:40.803774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:40.803774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:40.803774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.803774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:40.804213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:40.804780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:40.804780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:40.804780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:40.804780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:40.804780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:40.804780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:40.804780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:40.804780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:40.804780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.804780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:40.827024 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:40.827618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:40.827618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:40.827618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:34:40.827911 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:40.828107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.830445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:40.830445 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:40.830445 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:40.830445 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:40.830445 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:40.830445 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:40.830445 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.830445 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:40.830445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.830445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:40.830445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:40.830445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:40.830445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.830445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:40.830445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:40.830445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:40.830445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.830445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:40.830997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:40.830997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:40.830997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:40.830997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:40.830997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:40.830997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:40.830997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:40.830997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:40.830997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:40.830997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:40.830997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:40.830997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:40.830997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.830997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:40.830997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.831438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:40.831438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:40.831438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:40.831438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.831438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:40.831438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:40.831438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:34:40.831438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:40.831438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:40.831438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.831438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:40.831438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:40.831438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:40.831809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.831809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:40.831809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:40.831809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:40.831809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:40.831809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:40.831809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:40.831809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:40.831809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:40.831809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:40.831809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:40.831809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:40.831809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:40.831809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:40.832204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.832204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:40.832204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.832204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:40.832204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:40.832204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:40.832204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:40.832204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:40.832204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.832204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:40.832204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:40.832204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.832204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:40.832204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:40.832204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.832654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:40.832654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:40.832654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.832654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:40.832654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:40.832654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:34:40.832654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:40.832654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:40.832654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.832654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:40.832654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:40.832654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:40.832654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.832654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:40.832654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:40.832654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:40.832654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:40.832654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:40.832654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:40.833176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:40.833176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:40.833176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:40.833176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:40.833176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:40.833176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:40.833176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:40.833176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.833176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:40.838291 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:40.838727 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:40.838727 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:40.838727 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:40.838990 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:40.839188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.841484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:40.841484 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:40.841484 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:40.841484 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:40.841484 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:40.841484 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:40.841484 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.841484 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:40.841484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.841484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:40.841484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:40.841484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:40.841484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.841484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:40.841484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:40.841484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:40.841484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.841484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:40.842041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:40.842041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:40.842041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:40.842041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:40.842041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:40.842041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:40.842041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:40.842041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:40.842041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:40.842041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:40.842041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:40.842041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:40.842041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.842041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:40.842041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.842462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:40.842462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:40.842462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:40.842462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.842462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:40.842462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:40.842462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:40.842462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:40.842462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.842462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:40.842462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:40.842462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:40.842462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.842816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:40.842816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:40.842816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:40.842816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:40.842816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:40.842816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:40.842816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:40.842816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:40.842816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:40.842816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:40.842816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:40.842816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:40.842816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:40.843154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.843154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:40.843154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.843154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:40.843154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:40.843154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:40.843154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:40.843154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:40.843154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.843154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:40.843154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:40.843154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.843154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:40.843154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:40.843154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.843588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:40.843588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:40.843588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.843588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:40.843588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:40.843588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:40.843588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:40.843588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.843588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:40.843588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:40.843588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:40.843588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.843588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:40.843588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:40.843588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:40.843588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:40.843588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:40.843588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:40.843588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:40.844099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:40.844099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:40.844099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:40.844099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:40.844099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:40.844099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:40.844099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.844099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:40.856575 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:40.856844 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:40.857045 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:40.857280 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:40.857775 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:40.858057 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:40.858400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.860206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:40.860206 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:40.860206 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:40.860206 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:40.860206 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:40.860206 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:40.860206 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.860206 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:40.860206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.860206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:40.860206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:40.860206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:40.860206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.860206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:40.860206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:40.860674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.860674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:40.860674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:40.860674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:40.860674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.860674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:40.860674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:40.860674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:40.860674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:40.860674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:40.860674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:40.860674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:40.860674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:40.860674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:40.861055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:40.861055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:40.861055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:40.861055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:40.861055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.861055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:40.861055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.861055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:40.861055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:40.861055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:40.861055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:40.861055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:40.861055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.861055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:40.861453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:40.861983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:40.861983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:40.861983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:40.861983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:40.861983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:40.861983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:40.861983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:40.861983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:40.861983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:40.861983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:43.726425 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:43.728618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:43.728618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:43.728618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:43.773997 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:43.774335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:43.777523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:43.777523 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:43.777523 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:43.777523 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:43.777523 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:43.777523 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:43.777523 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:43.777523 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:43.777523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:43.777523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:43.777523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:43.777523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:43.777523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:43.777523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:43.777523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:43.779634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:43.779634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:43.779634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:43.779634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:43.779634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:43.779634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:43.779634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:43.779634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:43.779634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:43.779634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:43.779634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:43.779634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:43.779634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:43.779634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:43.780374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:43.780374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:43.780374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:43.780374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:43.780374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:43.780374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:43.780374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:43.780374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:43.780374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:43.780374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:43.780374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:43.780374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:43.780374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:43.780374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:43.780974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:43.781341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:43.781341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:43.781341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:43.781341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:43.781341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:43.781341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:43.781341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:43.781341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:43.781341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:45.798515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:45.800833 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:45.800833 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:45.800982 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:45.801138 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:45.801266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:45.801541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:45.801541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:45.801759 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:45.801759 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e367330-4323-4718-94c3-86375e03fbd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:45.802014 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cc40143-79da-4178-9539-6c8747af56a2, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:45.802014 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f424fd1-dda0-4816-aa5c-f5e45222260b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:45.802109 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:45.802154 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-346f9bd6-26ea-4f78-a9a4-201bc86255b6, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:45.802315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.804826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:45.804826 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:45.804826 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:45.804826 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:45.804826 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:45.804826 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:45.804826 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.804826 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:45.804826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.804826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:45.804826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:45.804826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:45.804826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.804826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:45.804826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:45.805579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.805579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:45.805579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:45.805579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:45.805579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.805579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:45.805579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:45.805579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:45.805579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:45.805579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:45.805579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:45.805579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:45.805579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:45.805579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:45.805938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:45.805938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:45.805938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:45.805938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:45.805938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.805938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:45.805938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.805938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:45.805938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:45.805938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:45.805938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:45.805938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:45.805938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.805938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:45.806791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:45.807417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:45.807417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:45.807417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:45.807417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:45.807417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:45.807417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:45.807417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:45.807417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:45.807417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.807417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:45.827759 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:45.829111 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:45.829488 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:45.829815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:34:45.830733 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:45.830959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.832423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:45.832423 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:45.832423 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:45.832423 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:45.832423 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:45.832423 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:45.832423 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.832423 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:45.833309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.833309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:45.833309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:45.833309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:45.833309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.833309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:45.833309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:45.833309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:45.833309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.833309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:45.833309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:45.833309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:45.833309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:45.833309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:45.833309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:45.833826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:45.833826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:45.833826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:45.833826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:45.833826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:45.833826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:45.833826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:45.833826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.833826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:45.836427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.836427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:45.836427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:45.836427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:45.836427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.836427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:45.836427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:45.836427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:34:45.836427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:45.836427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:45.836427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.836427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:45.837325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:45.837325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:45.837325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.837325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:45.837325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:45.837325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:45.837325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:45.837325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:45.837325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:45.837325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:45.837325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:45.837325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:45.837325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:45.837325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:45.837594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:45.837594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:45.837594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.837594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:45.838119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.838119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:45.838119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:45.838119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:45.838119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:45.838119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:45.838119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.838119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:45.838119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:45.838119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.838119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:45.838119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:45.838119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.838119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:45.838119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:45.838119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.838464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:45.838464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:45.838464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:34:45.838464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:45.838464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:45.838464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.838464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:45.838464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:45.838464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:45.838464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.838464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:45.838464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:45.838464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:45.838464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:45.838464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:45.838464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:45.838464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:45.838464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:45.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:45.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:45.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:45.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:45.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:45.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:45.839174 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:45.840263 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:45.840263 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:45.840263 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:45.840263 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:45.840582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.843770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:45.843770 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:45.843770 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:45.843770 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:45.843770 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:45.843770 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:45.843770 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.843770 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:45.843770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.843770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:45.843770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:45.843770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:45.843770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.843770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:45.843770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:45.843770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:45.843770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.843770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:45.844703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:45.844703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:45.844703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:45.844703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:45.844703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:45.844703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:45.844703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:45.844703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:45.844703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:45.844703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:45.844703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:45.844703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:45.844703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.844703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:45.844703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.845346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:45.845346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:45.845346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:45.845346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.845346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:45.845346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:45.845346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:45.845346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:45.845346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.845346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:45.845346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:45.845346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:45.845346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.845635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:45.845635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:45.845635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:45.845635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:45.845635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:45.845635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:45.845635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:45.845635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:45.845635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:45.845635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:45.845635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:45.845635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:45.845635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:45.845898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.845898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:45.845898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.845898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:45.845898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:45.845898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:45.845898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:45.845898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:45.845898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.845898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:45.845898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:45.845898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.845898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:45.845898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:45.845898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.846177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:45.846177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:45.846177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.846177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:45.846177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:45.846177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:45.846177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:45.846177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.846177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:45.846177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:45.846177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:45.846177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.846177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:45.846177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:45.846177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:45.846177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:45.846177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:45.846177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:45.846177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:45.846530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:45.846530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:45.846530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:45.846530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:45.846530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:45.846530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:45.846530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.846530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:45.857051 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:45.857509 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:45.857829 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:45.858110 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:45.858513 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:45.858698 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:45.859054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.859997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:45.859997 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:45.859997 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:45.859997 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:45.859997 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:45.859997 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:45.859997 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.859997 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:45.861115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.861115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:45.861115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:45.861115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:45.861115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.861115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:45.861115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:45.861115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.861115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:45.861115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:45.861115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:45.861115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.861638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:45.861638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:45.861638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:45.861638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:45.861638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:45.861638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:45.861638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:45.861638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:45.861638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:45.861638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:45.861638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:45.861638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:45.861638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:45.862246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.862246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:45.863695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.863695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:45.863695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:45.863695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:45.863695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:45.863695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:45.863695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.863695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:45.863695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:45.863695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.863695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:45.863695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:45.863695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.863695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:45.863695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:45.863695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:45.864045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:45.864045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:45.864045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.864045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:45.864045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:45.864045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:45.864045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:45.864045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:45.864045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:45.864045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:45.864045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:45.864045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:45.864045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:45.864045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:45.864045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:45.864045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:45.864045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:45.864045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:45.864045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:45.864411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:45.864411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:45.864411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:48.728478 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:48.731579 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:48.731717 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:48.732725 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:48.782438 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:48.782675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:48.783313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:48.783313 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:48.783313 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:48.783313 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:48.783313 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:48.783313 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:48.783313 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:48.783313 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:48.784374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.052 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:48.784374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:48.784374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:48.784374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:48.784374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:48.784374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:48.784374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:48.784374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:48.784374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:48.784374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:48.784374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:48.784374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:48.784893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:48.784893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:48.784893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:48.784893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:48.784893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:48.784893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:48.784893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:48.784893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:48.784893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:48.784893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:48.784893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:48.784893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:48.784893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:48.785317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:48.785317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:48.785579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:48.785579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:48.785579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:48.785579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:48.785579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:48.785579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:48.785579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:48.785579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:48.785579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:48.785579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:48.785579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:48.785579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:48.785579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:48.785579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:48.785579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:48.785579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:48.786062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:48.786062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:48.786062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:48.786062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:48.786062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:48.786062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:48.786062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:48.786062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:48.786062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:48.786062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:48.786062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:48.786062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:48.786062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:48.786062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:48.786062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:48.786062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:48.786062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:48.786062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:48.786062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:48.786605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:48.786605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:48.786605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:50.799538 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:50.800658 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:50.800658 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:50.800963 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:50.801137 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:50.801336 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:50.801438 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:50.801586 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e367330-4323-4718-94c3-86375e03fbd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:50.801733 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f424fd1-dda0-4816-aa5c-f5e45222260b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:50.801867 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:50.802018 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:50.802156 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-41a5315e-2c61-493c-a648-d17e398b7e74, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:50.802309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.802936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:50.802936 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:50.802936 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:50.802936 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:50.802936 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:50.802936 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:50.802936 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.802936 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:50.803864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.803864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:50.803864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:50.803864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:50.803864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.803864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:50.803864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:50.803864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.803864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:50.803864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:50.803864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:50.803864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.804165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:50.804165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:50.804165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:50.804165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:50.804165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:50.804165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:50.804165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:50.804165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:50.804165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:50.804165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:50.804165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:50.804165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:50.804165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:50.804425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.804425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:50.805076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.805076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:50.805076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:50.805076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:50.805076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:50.805076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:50.805076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.805076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:50.805076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:50.805076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.805076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:50.805076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:50.805076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.805076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:50.805076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:50.805076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.805401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:50.805401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:50.805401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.805401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:50.805401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:50.805401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:50.805401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.805401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:50.805401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:50.805401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:50.805401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:50.805401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:50.805401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:50.805401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:50.805401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:50.805401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:50.805401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:50.805401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:50.805401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:50.805700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:50.805700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.805700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:50.828272 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:50.829589 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:50.829687 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:50.829781 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:34:50.830188 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:50.830279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.830673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:50.830673 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:50.830673 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:50.830673 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:50.830673 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:50.830673 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:50.830673 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.830673 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:50.831206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.831206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:50.831206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:50.831206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:50.831206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.831206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:50.831206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:50.831206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:50.831206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.831206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:50.831206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:50.831206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:50.831206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:50.831206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:50.831206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:50.831744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:50.831744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:50.831744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:50.831744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:50.831744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:50.831744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:50.831744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:50.831744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.831744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:50.832276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.832276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:50.832276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:50.832276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:50.832276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.832276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:50.832276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:50.832276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:34:50.832276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:50.832276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:50.832276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.832276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:50.832525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:50.832525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:50.832525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.832525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:50.832525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:50.832525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:50.832525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:50.832525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:50.832525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:50.832525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:50.832525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:50.832525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:50.832525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:50.832525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:50.832744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:50.832744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:50.832744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.832744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:50.834140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.834140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:50.834140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:50.834140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:50.834140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:50.834140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:50.834140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.834140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:50.834140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:50.834140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.834140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:50.834140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:50.834140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.834140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:50.834140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:50.834140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.834454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:50.834454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:50.834454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:34:50.834454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:50.834454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:50.834454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.834454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:50.834454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:50.834454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:50.834454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.834454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:50.834454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:50.834454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:50.834454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:50.834454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:50.834454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:50.834454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:50.834454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:50.834744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:50.834744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:50.834744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:50.834744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:50.834744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:50.834744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.834744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:50.839676 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:50.840011 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:50.840104 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:50.840197 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:50.840496 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:50.840662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.841260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:50.841260 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:50.841260 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:50.841260 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:50.841260 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:50.841260 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:50.841260 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.841260 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:50.842566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.842566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:50.842566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:50.842566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:50.842566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.842566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:50.842566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:50.842566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:50.842566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.842566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:50.842566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:50.842566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:50.842566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:50.842566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:50.842566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:50.842903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:50.842903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:50.842903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:50.842903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:50.842903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:50.842903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:50.842903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:50.842903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.842903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:50.843152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.843152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:50.843152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:50.843152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:50.843152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.843152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:50.843152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:50.843152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:50.843152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:50.843152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.843152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:50.843347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:50.843347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:50.843347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.843347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:50.843347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:50.843347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:50.843347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:50.843347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:50.843347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:50.843347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:50.843347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:50.843347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:50.843347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:50.843347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:50.843571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:50.843571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:50.843571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.843571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:50.844975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.844975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:50.844975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:50.844975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:50.844975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:50.844975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:50.844975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.844975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:50.844975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:50.844975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.844975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:50.844975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:50.844975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.844975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:50.844975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:50.844975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.844975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:50.845331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:50.845331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:50.845331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:50.845331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.845331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:50.845331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:50.845331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:50.845331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.845331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:50.845331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:50.845331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:50.845331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:50.845331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:50.845331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:50.845331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:50.845331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:50.845331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:50.845331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:50.845331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:50.845692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:50.845692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:50.845692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.845692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:50.857584 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:50.860003 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:50.860087 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:50.860222 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:50.860491 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:50.860674 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:50.860886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.861947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:50.861947 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:50.861947 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:50.861947 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:50.861947 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:50.861947 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:50.861947 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.861947 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:50.862532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.862532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:50.862532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:50.862532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:50.862532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.862532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:50.862532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:50.862532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.862532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:50.862532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:50.862532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:50.862532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.862761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:50.862761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:50.862761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:50.862761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:50.862761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:50.862761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:50.862761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:50.862761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:50.862761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:50.862761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:50.862761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:50.862761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:50.862761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:50.862983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.862983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:50.863179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.863179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:50.863179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:50.863179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:50.863179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:50.863179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:50.863179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.863179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:50.863179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:50.863179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.863179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:50.863179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:50.863179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.863179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:50.863179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:50.863179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:50.863543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:50.863543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:50.863543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.863543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:50.863543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:50.863543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:50.863543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:50.863543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:50.863543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:50.863543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:50.863543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:50.863543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:50.863543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:50.863543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:50.863543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:50.863543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:50.863543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:50.863543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:50.863543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:50.863825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:50.863825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:50.863825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:53.732445 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:53.732971 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:53.732971 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:53.733541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:53.734689 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=239 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:34:53.764004 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:53.764178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:53.764664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:53.764664 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:53.764664 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:53.764664 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:53.764664 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:53.764664 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:53.764664 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:53.764664 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:53.765311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:53.765311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:53.765311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:53.765311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:53.765311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:53.765311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:53.765311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:53.765311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:53.765311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:53.765311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:53.765311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:53.765311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:53.765725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:53.765725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:53.765725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:53.765725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:53.765725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:53.765725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:53.765725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:53.765725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:53.765725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:53.765725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:53.765725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:53.765725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:53.765725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:53.766032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:53.766032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:53.766199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:53.766199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:53.766199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:53.766199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:53.766199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:53.766199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:53.766199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:53.766199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:53.766199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:53.766199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:53.766199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:53.766199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:53.766199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:53.766199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:53.766199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:53.766199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:53.766792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:53.766792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:53.766792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:53.766792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:53.766792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:53.766792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:53.766792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:53.766792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:53.766792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:53.766792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:53.766792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:53.766792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:53.766792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:53.766792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:53.766792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:53.766792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:53.766792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:53.766792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:53.766792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:53.767518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:53.767518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:53.767518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:55.800048 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:55.800658 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:55.800658 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:55.801341 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:55.801470 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:55.801582 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:55.801690 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:55.801779 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e367330-4323-4718-94c3-86375e03fbd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:55.801873 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f424fd1-dda0-4816-aa5c-f5e45222260b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:55.801980 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:55.802083 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:55.802187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.802727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:55.802727 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:55.802727 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:55.802727 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:55.802727 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:55.802727 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:55.802727 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.802727 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:55.803358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.803358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:55.803358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:55.803358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:55.803358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.803358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:55.803358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:55.803358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.803358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:55.803358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:55.803358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:55.803358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.803770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:55.803770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:55.803770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:55.803770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:55.803770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:55.803770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:55.803770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:55.803770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:55.803770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:55.803770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:55.803770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:55.803770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:55.803770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:55.804154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.804154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:55.804154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.804154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:55.804154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:55.804154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:55.804154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:55.804154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:55.804154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.804154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:55.804154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:55.804154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.804154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:55.804154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:55.804154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.804617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:55.804617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:55.804617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.804617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:34:55.804617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:55.804617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.804617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:55.804617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:55.804617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:55.804617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.804617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:55.804617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:55.804617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:55.804617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:55.804617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:55.804617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:55.804617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:55.804617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:55.804617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:55.805136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:55.805136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:55.805136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:55.805136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:55.805136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.805136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:55.829696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:55.830074 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:55.830170 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:55.830275 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:34:55.830631 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:55.830822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.831219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:55.831219 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:55.831219 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:55.831219 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:55.831219 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:55.831219 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:55.831219 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.831219 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:55.831753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.831753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:55.831753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:55.831753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:55.831753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.831753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:55.831753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:55.831753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:55.831753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.831753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:55.831753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:55.831753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:55.831753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:55.831753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:55.831753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:55.832282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:55.832282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:55.832282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:55.832282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:55.832282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:55.832282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:55.832282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:55.832282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.832282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:55.832548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.832548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:55.832548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:55.832548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:55.832548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.832548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:55.832548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:55.832548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:34:55.832548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:55.832548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:55.832548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.832548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:55.832878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:55.832878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:55.832878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.832878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:55.832878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:55.832878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:55.832878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:55.832878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:55.832878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:55.832878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:55.832878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:55.832878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:55.832878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:55.832878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:55.833290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:55.833290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:55.833290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.833290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:55.833290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.833290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:55.833290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:55.833290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:55.833290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:55.833290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:55.833290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.833290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:55.833290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:55.833290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.833689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:55.833689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:55.833689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.833689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:55.833689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:55.833689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.833689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:34:55.833689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:34:55.833689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:34:55.833689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:55.833689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:55.833689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.833689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:55.833689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:55.833689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:55.833689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.833689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:55.833689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:55.833689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:55.834210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:55.834210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:55.834210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:55.834210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:55.834210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:55.834210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:55.834210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:55.834210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:55.834210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:55.834210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:55.834210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.834210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:55.840079 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:55.840541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:55.840634 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:55.840761 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:55.841090 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:55.841274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.841689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:55.841689 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:55.841689 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:55.841689 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:55.841689 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:55.841689 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:55.841689 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.841689 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:55.842198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.842198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:34:55.842198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:55.842198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:55.842198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.842198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:55.842198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:55.842198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:34:55.842198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.842198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:34:55.842198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:55.842198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:34:55.842198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:55.842198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:34:55.842198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:34:55.842668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:55.842668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:34:55.842668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:55.842668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:34:55.842668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:34:55.842668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:55.842668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:34:55.842668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.842668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:34:55.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:55.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:55.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:55.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:55.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:55.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:55.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:55.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:55.843304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:55.843304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:55.843304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.843304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:55.843304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:55.843304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:55.843304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:55.843304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:55.843304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:55.843304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:55.843304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:55.843304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:55.843304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:55.843304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:55.843690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:55.843690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:55.843690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.843690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:55.843690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.843690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:55.843690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:55.843690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:55.843690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:55.843690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:55.843690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.843690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:55.843690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:55.843690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.844105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:55.844105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:55.844105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.844105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:55.844105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:55.844105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.844105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:34:55.844105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:34:55.844105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:34:55.844105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:34:55.844105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.844105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:55.844105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:55.844105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:55.844105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.844105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:55.844105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:55.844105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:55.844105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:55.844730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:55.844730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:55.844730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:55.844730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:55.844730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:55.844730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:55.844730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:55.844730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:55.844730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:55.844730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.844730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:55.860096 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:55.861607 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:55.861712 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:55.861870 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:55.862195 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:55.862427 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:55.862633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.864314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:55.864314 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:55.864314 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:55.864314 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:55.864314 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:55.864314 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:55.864314 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.864314 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:55.864946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.864946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:55.864946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:55.864946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:55.864946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.864946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:55.864946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:55.864946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.864946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:55.864946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:55.864946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:55.864946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.865199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:55.865199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:55.865199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:55.865199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:55.865199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:55.865199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:55.865199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:55.865199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:55.865199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:55.865199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:55.865199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:55.865199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:55.865199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:55.865446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.865446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:55.865665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.865665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:55.865665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:55.865665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:55.865665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:55.865665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:55.865665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.865665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:55.865665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:55.865665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.865665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:55.865665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:55.865665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.865665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:55.865665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:55.865665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:55.866062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:34:55.866062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:55.866062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.866062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:55.866062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:55.866062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:55.866062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:55.866062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:55.866062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:55.866062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:55.866062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:55.866062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:55.866062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:55.866062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:55.866062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:55.866062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:55.866062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:55.866062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:55.866062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:55.866428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:55.866428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:55.866428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:34:58.732931 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:34:58.733327 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:34:58.733448 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:34:58.734279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:34:58.771110 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:34:58.771308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:58.771763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:34:58.771763 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:58.771763 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:34:58.771763 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:34:58.771763 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:58.771763 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:34:58.771763 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:58.771763 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:34:58.772424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:58.772424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:34:58.772424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:58.772424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:34:58.772424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:58.772424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:58.772424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:58.772424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:58.772424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:58.772424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:58.772424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:34:58.772424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:58.772758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:34:58.772758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:58.772758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:34:58.772758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:58.772758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:34:58.772758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:34:58.772758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:58.772758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:34:58.772758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:58.772758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:34:58.772758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:34:58.772758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:58.772758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:34:58.773060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:58.773060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:34:58.775028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:58.775028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:34:58.775028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:34:58.775028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:34:58.775028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:34:58.775028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:34:58.775028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:58.775028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:34:58.775028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:34:58.775028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:58.775028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:34:58.775028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:34:58.775028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:34:58.775028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:34:58.775028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:34:58.775028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:34:58.775461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:34:58.775461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:34:58.775461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:58.775461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:58.775461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:34:58.775461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:34:58.775461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:34:58.775461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:34:58.775461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:34:58.775461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:34:58.775461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:34:58.775461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:34:58.775461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:34:58.775461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:34:58.775461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:34:58.775461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:34:58.775461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:34:58.775461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:34:58.775461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:34:58.775928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:34:58.775928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:34:58.775928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:00.800432 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:00.801047 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:00.801164 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:00.801975 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:00.802110 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:00.802220 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:00.802351 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:00.802494 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e367330-4323-4718-94c3-86375e03fbd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:00.802616 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f424fd1-dda0-4816-aa5c-f5e45222260b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:00.802748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:00.802875 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:00.803022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.805392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:00.805392 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:00.805392 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:00.805392 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:00.805392 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:00.805392 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:00.805392 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.805392 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:00.806048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.806048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:00.806048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:00.806048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:00.806048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.806048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:00.806048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:00.806048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.806048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:00.806048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:00.806048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:00.806048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.806297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:00.806297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:00.806297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:00.806297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:00.806297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:00.806297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:00.806297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:00.806297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:00.806297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:00.806297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:00.806297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:00.806297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:00.806297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:00.806592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.806592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:00.806833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.806833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:00.806833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:00.806833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:00.806833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:00.806833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:00.806833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.806833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:00.806833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:00.806833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.806833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:00.806833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:00.806833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.806833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:00.806833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:00.806833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.807087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:00.807087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:00.807087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.807087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:00.807087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:00.807087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:00.807087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.807087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:00.807087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:00.807087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:00.807087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:00.807087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:00.807087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:00.807087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:00.807087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:00.807087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:00.807087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:00.807087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:00.807087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:00.807398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:00.807398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.807398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:00.830422 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:00.830558 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:00.830665 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:00.830781 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:35:00.834200 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:00.834200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.834540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:00.834540 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:00.834540 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:00.834540 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:00.834540 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:00.834540 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:00.834540 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.834540 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:00.835124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.835124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:00.835124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:00.835124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:00.835124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.835124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:00.835124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:00.835124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:00.835124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.835124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:00.835124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:00.835124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:00.835124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:00.835124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:00.835124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:00.835398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:00.835398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:00.835398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:00.835398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:00.835398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:00.835398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:00.835398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:00.835398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.835398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:00.835897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.835897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:00.835897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:00.835897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:00.835897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.835897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:00.835897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:00.835897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:35:00.835897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:00.835897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:00.835897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.835897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:00.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:00.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:00.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:00.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:00.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:00.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:00.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:00.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:00.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:00.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:00.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:00.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:00.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:00.836426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:00.836426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:00.836426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.836426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:00.836857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.836857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:00.836857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:00.836857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:00.836857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:00.836857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:00.836857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.836857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:00.836857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:00.836857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.836857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:00.836857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:00.836857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.836857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:00.836857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:00.836857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.837337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:00.837337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:00.837337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:35:00.837337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:00.837337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:00.837337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.837337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:00.837337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:00.837337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:00.837337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.837337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:00.837337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:00.837337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:00.837337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:00.837337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:00.837337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:00.837337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:00.837337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:00.837811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:00.837811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:00.837811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:00.837811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:00.837811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:00.837811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.837811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:00.840405 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:00.844104 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:00.844104 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:00.844104 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:00.844301 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:00.844517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.845608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:00.845608 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:00.845608 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:00.845608 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:00.845608 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:00.845608 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:00.845608 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.845608 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:00.846465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.846465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:00.846465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:00.846465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:00.846465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.846465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:00.846465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:00.846465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:00.846465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.846465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:00.846465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:00.846465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:00.846465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:00.846465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:00.846465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:00.846742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:00.846742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:00.846742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:00.846742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:00.846742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:00.846742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:00.846742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:00.846742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.846742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:00.847135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.847135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:00.847135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:00.847135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:00.847135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.847135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:00.847135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:00.847135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:00.847135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:00.847135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.847135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:00.847420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:00.847420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:00.847420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.847420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:00.847420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:00.847420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:00.847420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:00.847420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:00.847420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:00.847420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:00.847420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:00.847420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:00.847420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:00.847420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:00.847668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:00.847668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:00.847668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.847668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:00.847973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.847973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:00.847973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:00.847973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:00.847973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:00.847973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:00.847973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.847973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:00.847973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:00.847973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.847973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:00.847973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:00.847973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.847973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:00.847973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:00.847973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.847973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:00.848355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:00.848355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:00.848355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:00.848355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.848355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:00.848355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:00.848355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:00.848355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.848355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:00.848355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:00.848355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:00.848355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:00.848355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:00.848355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:00.848355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:00.848355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:00.848355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:00.848355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:00.848355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:00.848710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:00.848710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:00.848710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.848710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:00.861656 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:00.862006 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:00.862105 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:00.862261 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:00.862497 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:00.862705 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:00.862879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.863301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:00.863301 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:00.863301 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:00.863301 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:00.863301 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:00.863301 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:00.863301 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.863301 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:00.863936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.863936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:00.863936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:00.863936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:00.863936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.863936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:00.863936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:00.863936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.863936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:00.863936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:00.863936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:00.863936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.864327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:00.864327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:00.864327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:00.864327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:00.864327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:00.864327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:00.864327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:00.864327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:00.864327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:00.864327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:00.864327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:00.864327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:00.864327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:00.864743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.864743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:00.864743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.864743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:00.864743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:00.864743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:00.864743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:00.864743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:00.864743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.864743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:00.864743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:00.864743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.864743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:00.864743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:00.864743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.865135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:00.865135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:00.865135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:00.865135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:00.865135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:00.865135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.865135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:00.865135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:00.865135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:00.865135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:00.865135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:00.865135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:00.865135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:00.865135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:00.865135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:00.865135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:00.865135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:00.865135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:00.865135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:00.865876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:00.865876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:00.865876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:00.865876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:00.865876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:00.865876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:03.733354 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:03.734306 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:03.734754 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:03.735657 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:03.790892 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:03.791089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:03.791530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:03.791530 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:03.791530 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:03.791530 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:03.791530 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:03.791530 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:03.791530 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:03.791530 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:03.792183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.057 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:03.792183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:03.792183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:03.792183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:03.792183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:03.792183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:03.792183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:03.792183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:03.792183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:03.792183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:03.792183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:03.792183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:03.792491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:03.792491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:03.792491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:03.792491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:03.792491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:03.792491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:03.792491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:03.792491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:03.792491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:03.792491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:03.792491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:03.792491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:03.792491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:03.792714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:03.792714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:03.793518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:03.793518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:03.793518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:03.793518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:03.793518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:03.793518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:03.793518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:03.793518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:03.793518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:03.793518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:03.793518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:03.793518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:03.793518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:03.793518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:03.793518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:03.793518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:03.793813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:03.793813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:03.793813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:03.793813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:03.793813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:03.793813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:03.793813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:03.793813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:03.793813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:03.793813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:03.793813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:03.793813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:03.793813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:03.793813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:03.793813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:03.793813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:03.793813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:03.793813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:03.793813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:03.794223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:03.794223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:03.794223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:05.801099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:05.802631 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:05.802631 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:05.802819 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:05.802998 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:05.803154 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:05.803276 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e367330-4323-4718-94c3-86375e03fbd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:05.803396 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f424fd1-dda0-4816-aa5c-f5e45222260b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:05.803499 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:05.803605 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:05.803743 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:05.803868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:05.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:05.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:05.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:05.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:05.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:05.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.804427 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:05.806713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.806713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:05.806713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:05.806713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:05.806713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.806713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:05.806713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:05.806713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.806713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:05.806713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:05.806713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:05.806713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.807141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:05.807141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:05.807141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:05.807141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:05.807141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:05.807141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:05.807141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:05.807141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:05.807141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:05.807141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:05.807141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:05.807141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:05.807141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:05.807576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.807576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:05.807576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.807576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:05.807576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:05.807576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:05.807576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:05.807576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:05.807576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.807576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:05.807576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:05.807576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.807576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:05.807576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:05.807576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.808123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:05.808123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:05.808123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.808123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:05.808123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:05.808123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.808123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:05.808123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:05.808123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:05.808123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.808123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:05.808123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:05.808123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:05.808123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:05.808123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:05.808123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:05.808123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:05.808123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:05.808123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:05.808803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:05.808803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:05.808803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:05.808803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:05.808803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.808803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:05.830495 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:05.831347 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:05.831347 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:05.831347 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:35:05.831592 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:05.831812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.834578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:05.834578 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:05.834578 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:05.834578 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:05.834578 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:05.834578 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:05.834578 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.834578 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:05.834578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.834578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:05.834578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:05.834578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:05.834578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.834578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:05.834578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:05.834578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:05.834578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.834578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:05.835035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:05.835035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:05.835035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:05.835035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:05.835035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:05.835035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:05.835035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:05.835035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:05.835035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:05.835035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:05.835035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:05.835035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:05.835035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.835035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:05.835035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.835424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:05.835424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:05.835424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:05.835424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.835424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:05.835424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:05.835424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:35:05.835424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:05.835424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:05.835424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.835424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:05.835424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:05.835424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:05.835774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.835774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:05.835774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:05.835774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:05.835774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:05.835774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:05.835774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:05.835774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:05.835774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:05.835774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:05.835774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:05.835774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:05.835774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:05.835774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:05.836076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.836076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:05.836076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.836076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:05.836076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:05.836076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:05.836076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:05.836076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:05.836076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.836076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:05.836076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:05.836076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.836076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:05.836076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:05.836076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.836371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:05.836371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:05.836371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.836371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:05.836371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:05.836371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:35:05.836371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:05.836371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:05.836371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.836371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:05.836371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:05.836371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:05.836371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.836371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:05.836371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:05.836371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:05.836371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:05.836371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:05.836371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:05.836825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:05.836825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:05.836825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:05.836825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:05.836825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:05.836825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:05.836825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:05.836825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.836825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:05.840785 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:05.841071 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:05.841264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:05.841465 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:05.841822 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:05.842037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.844349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:05.844349 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:05.844349 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:05.844349 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:05.844349 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:05.844349 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:05.844349 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.844349 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:05.844349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.844349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:05.844349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:05.844349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:05.844349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.844349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:05.844349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:05.844349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:05.844349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.844349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:05.844860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:05.844860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:05.844860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:05.844860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:05.844860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:05.844860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:05.844860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:05.844860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:05.844860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:05.844860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:05.844860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:05.844860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:05.844860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.844860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:05.844860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.845239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:05.845239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:05.845239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:05.845239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.845239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:05.845239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:05.845239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:05.845239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:05.845239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.845239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:05.845239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:05.845239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:05.845239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.845485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:05.845485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:05.845485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:05.845485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:05.845485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:05.845485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:05.845485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:05.845485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:05.845485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:05.845485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:05.845485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:05.845485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:05.845485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:05.845793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.845793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:05.845793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.845793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:05.845793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:05.845793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:05.845793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:05.845793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:05.845793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.845793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:05.845793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:05.845793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.845793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:05.845793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:05.845793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.846162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:05.846162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:05.846162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.846162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:05.846162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:05.846162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:05.846162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:05.846162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.846162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:05.846162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:05.846162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:05.846162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.846162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:05.846162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:05.846162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:05.846162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:05.846162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:05.846162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:05.846162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:05.846523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:05.846523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:05.846523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:05.846523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:05.846523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:05.846523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:05.846523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.846523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:05.862016 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:05.862653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:05.862653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:05.862653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:05.862993 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:05.863193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:05.863426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.865290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:05.865290 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:05.865290 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:05.865290 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:05.865290 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:05.865290 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:05.865290 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.865290 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:05.865290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.865290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:05.865290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:05.865290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:05.865290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.865290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:05.865290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:05.865635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.865635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:05.865635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:05.865635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:05.865635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.865635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:05.865635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:05.865635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:05.865635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:05.865635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:05.865635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:05.865635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:05.865635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:05.865635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:05.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:05.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:05.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:05.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:05.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:05.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:05.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:05.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:05.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:05.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:05.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:05.866263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:05.866678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:05.866678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:05.866678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:05.866678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:05.866678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:05.866678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:05.866678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:05.866678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:05.866678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:05.866678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:08.733729 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:08.735611 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:08.735611 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:08.735611 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:08.764152 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:08.764347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:08.766161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:08.766161 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:08.766161 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:08.766161 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:08.766161 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:08.766161 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:08.766161 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:08.766161 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:08.766161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:08.766161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:08.766161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:08.766161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:08.766161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:08.766161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:08.766161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:08.766703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:08.766703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:08.766703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:08.766703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:08.766703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:08.766703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:08.766703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:08.766703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:08.766703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:08.766703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:08.766703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:08.766703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:08.766703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:08.766703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:08.767086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:08.767086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:08.767086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:08.767086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:08.767086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:08.767086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:08.767086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:08.767086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:08.767086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:08.767086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:08.767086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:08.767086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:08.767086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:08.767086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:08.767497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:08.768064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:08.768064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:08.768064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:08.768064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:08.768064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:08.768064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:08.768064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:08.768064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:08.768064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:10.801984 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:10.803289 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:10.803289 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:10.803658 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:10.803837 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:10.803966 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:10.804077 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e367330-4323-4718-94c3-86375e03fbd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:10.804178 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f424fd1-dda0-4816-aa5c-f5e45222260b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:10.804318 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:10.804423 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:10.804526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:10.804628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.806656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:10.806656 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:10.806656 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:10.806656 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:10.806656 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:10.806656 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:10.806656 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.806656 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:10.806656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.806656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:10.806656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:10.806656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:10.806656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.806656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:10.806656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:10.807177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.807177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:10.807177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:10.807177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:10.807177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.807177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:10.807177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:10.807177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:10.807177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:10.807177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:10.807177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:10.807177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:10.807177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:10.807177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:10.807577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:10.807577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:10.807577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:10.807577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:10.807577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.807577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:10.807577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.807577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:10.807577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:10.807577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:10.807577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:10.807577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:10.807577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.807577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:10.807969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:10.808539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:10.808539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:10.808539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:10.808539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:10.808539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:10.808539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:10.808539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:10.808539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:10.808539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.808539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:10.830988 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:10.831542 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:10.831542 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:10.831542 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:35:10.831765 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:10.831932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.832307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:10.832307 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:10.832307 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:10.832307 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:10.832307 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:10.832307 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:10.832307 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.832307 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:10.834418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.834418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:10.834418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:10.834418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:10.834418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.834418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:10.834418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:10.834418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:10.834418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.834418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:10.834418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:10.834418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:10.834418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:10.834418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:10.834418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:10.834913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:10.834913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:10.834913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:10.834913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:10.834913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:10.834913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:10.834913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:10.834913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.834913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:10.834913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.834913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:10.834913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:10.834913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:10.834913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.835324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:10.835324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:10.835324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:35:10.835324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:10.835324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:10.835324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.835324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:10.835324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:10.835324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:10.835324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.835324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:10.835324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:10.835324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:10.835669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:10.835669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:10.835669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:10.835669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:10.835669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:10.835669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:10.835669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:10.835669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:10.835669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:10.835669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:10.835669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.835669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:10.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:10.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:10.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:10.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:10.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:10.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:10.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:10.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:10.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:10.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:10.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:10.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.836472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:10.836472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:10.836472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:35:10.836472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:10.836472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:10.836472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.836472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:10.836472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:10.836472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:10.836472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.836472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:10.836472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:10.836472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:10.836472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:10.836472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:10.836472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:10.836472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:10.836472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:10.836992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:10.836992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:10.836992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:10.836992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:10.836992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:10.836992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.836992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:10.841108 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:10.841366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:10.841541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:10.841720 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:10.842042 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:10.842236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.844315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:10.844315 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:10.844315 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:10.844315 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:10.844315 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:10.844315 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:10.844315 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.844315 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:10.844315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.844315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:10.844315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:10.844315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:10.844315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.844315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:10.844315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:10.844315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:10.844315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.844315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:10.844847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:10.844847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:10.844847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:10.844847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:10.844847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:10.844847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:10.844847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:10.844847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:10.844847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:10.844847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:10.844847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:10.844847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:10.844847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.844847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:10.844847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.845372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:10.845372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:10.845372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:10.845372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.845372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:10.845372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:10.845372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:10.845372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:10.845372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.845372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:10.845372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:10.845372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:10.845372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.845739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:10.845739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:10.845739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:10.845739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:10.845739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:10.845739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:10.845739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:10.845739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:10.845739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:10.845739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:10.845739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:10.845739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:10.845739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:10.846125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.846125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:10.846125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.846125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:10.846125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:10.846125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:10.846125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:10.846125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:10.846125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.846125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:10.846125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:10.846125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.846125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:10.846125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:10.846125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.846593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:10.846593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:10.846593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.846593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:10.846593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:10.846593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:10.846593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:10.846593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.846593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:10.846593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:10.846593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:10.846593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.846593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:10.846593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:10.846593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:10.846593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:10.846593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:10.846593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:10.846593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:10.847124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:10.847124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:10.847124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:10.847124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:10.847124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:10.847124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:10.847124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.847124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:10.862419 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:10.862959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:10.862959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:10.862959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:10.863212 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:10.863437 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:10.863670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:10.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:10.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:10.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:10.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:10.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:10.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:10.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:10.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:10.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:10.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:10.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:10.865785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.865785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:10.865785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:10.865785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:10.865785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.865785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:10.865785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:10.865785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:10.865785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:10.865785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:10.865785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:10.865785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:10.865785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:10.865785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:10.866039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:10.866039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:10.866039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:10.866039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:10.866039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.866039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:10.866039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.866039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:10.866039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:10.866039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:10.866039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:10.866039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:10.866039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.866039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:10.866302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:10.866665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:10.866665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:10.866665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:10.866665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:10.866665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:10.866665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:10.866665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:10.866665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:10.866665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:10.866665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:13.734521 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:13.735288 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:13.735288 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:13.738027 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:13.786726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:13.786992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:13.787525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:13.787525 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:13.787525 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:13.787525 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:13.787525 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:13.787525 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:13.787525 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:13.787525 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:13.788134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.053 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:13.788134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:13.788134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:13.788134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:13.788134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:13.788134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:13.788134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:13.788134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:13.788134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:13.788134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:13.788134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:13.788134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:13.788570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:13.788570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:13.788570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:13.788570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:13.788570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:13.788570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:13.788570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:13.788570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:13.788570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:13.788570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:13.788570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:13.788570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:13.788570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:13.788984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:13.788984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:13.788984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:13.788984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:13.788984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:13.788984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:13.788984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:13.788984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:13.788984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:13.788984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:13.788984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:13.788984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:13.788984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:13.788984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:13.788984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:13.789462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:13.789462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:13.789462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:13.789462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:13.789462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:13.789462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:13.789462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:13.789462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:13.789462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:13.789462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:13.789462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:13.789462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:13.789462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:13.789462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:13.789462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:13.789462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:13.789462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:13.789462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:13.789462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:13.790027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:13.790027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:13.790027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:13.790027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:13.790027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:13.790027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:15.802821 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:15.803482 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:15.803482 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:15.804126 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:15.804279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:15.804397 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:15.804495 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e367330-4323-4718-94c3-86375e03fbd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:15.804584 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f424fd1-dda0-4816-aa5c-f5e45222260b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:15.804681 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:15.804838 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:15.804961 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:15.805057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.805573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:15.805573 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:15.805573 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:15.805573 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:15.805573 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:15.805573 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:15.805573 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.805573 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:15.806170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.806170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:15.806170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:15.806170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:15.806170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.806170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:15.806170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:15.806170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.806170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:15.806170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:15.806170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:15.806170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.806405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:15.806405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:15.806405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:15.806405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:15.806405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:15.806405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:15.806405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:15.806405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:15.806405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:15.806405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:15.806405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:15.806405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:15.806405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:15.806594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.806594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:15.806856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.806856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:15.806856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:15.806856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:15.806856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:15.806856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:15.806856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.806856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:15.806856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:15.806856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.806856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:15.806856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:15.806856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.806856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:15.806856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:15.806856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:15.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:15.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:15.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:15.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:15.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:15.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:15.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:15.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:15.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:15.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:15.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:15.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:15.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:15.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:15.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:15.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:15.807806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:15.807806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.807806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:15.831443 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:15.831687 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:15.831788 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:15.831912 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:35:15.832172 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:15.832346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.832734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:15.832734 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:15.832734 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:15.832734 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:15.832734 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:15.832734 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:15.832734 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.832734 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:15.833345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.833345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:15.833345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:15.833345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:15.833345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.833345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:15.833345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:15.833345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:15.833345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.833345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:15.833345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:15.833345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:15.833345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:15.833345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:15.833345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:15.833615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:15.833615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:15.833615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:15.833615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:15.833615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:15.833615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:15.833615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:15.833615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.833615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:15.833880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.833880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:15.833880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:15.833880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:15.833880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.833880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:15.833880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:15.833880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:35:15.833880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:15.833880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:15.833880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.833880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:15.834104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:15.834104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:15.834104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.834104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:15.834104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:15.834104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:15.834104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:15.834104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:15.834104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:15.834104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:15.834104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:15.834104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:15.834104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:15.834104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:15.834321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:15.834321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:15.834321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.834321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:15.834561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.834561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:15.834561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:15.834561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:15.834561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:15.834561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:15.834561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.834561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:15.834561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:15.834561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.834561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:15.834561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:15.834561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.834561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:15.834561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:15.834561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.834961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:15.834961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:15.834961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:35:15.834961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:15.834961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:15.834961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.834961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:15.834961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:15.834961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:15.834961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.834961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:15.834961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:15.834961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:15.834961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:15.834961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:15.834961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:15.834961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:15.834961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:15.835466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:15.835466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:15.835466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:15.835466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:15.835466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:15.835466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.835466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:15.841399 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:15.841590 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:15.841679 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:15.841769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:15.842011 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:15.842170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.842497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:15.842497 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:15.842497 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:15.842497 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:15.842497 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:15.842497 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:15.842497 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.842497 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:15.843008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.843008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:15.843008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:15.843008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:15.843008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.843008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:15.843008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:15.843008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:15.843008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.843008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:15.843008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:15.843008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:15.843008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:15.843008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:15.843008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:15.843333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:15.843333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:15.843333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:15.843333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:15.843333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:15.843333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:15.843333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:15.843333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.843333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:15.843559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.843559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:15.843559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:15.843559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:15.843559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.843559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:15.843559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:15.843559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:15.843559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:15.843559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.843559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:15.843756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:15.843756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:15.843756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.843756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:15.843756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:15.843756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:15.843756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:15.843756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:15.843756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:15.843756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:15.843756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:15.843756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:15.843756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:15.843756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:15.843972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:15.843972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:15.843972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.843972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:15.844267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.844267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:15.844267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:15.844267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:15.844267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:15.844267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:15.844267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.844267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:15.844267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:15.844267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.844267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:15.844267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:15.844267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.844267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:15.844267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:15.844267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.844267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:15.844650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:15.844650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:15.844650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:15.844650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.844650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:15.844650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:15.844650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:15.844650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.844650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:15.844650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:15.844650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:15.844650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:15.844650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:15.844650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:15.844650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:15.844650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:15.844650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:15.844650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:15.844650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:15.845156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:15.845156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:15.845156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.845156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:15.862863 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:15.863148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:15.863281 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:15.863451 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:15.863731 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:15.863970 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:15.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.864597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:15.864597 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:15.864597 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:15.864597 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:15.864597 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:15.864597 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:15.864597 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.864597 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:15.865237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.865237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:15.865237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:15.865237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:15.865237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.865237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:15.865237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:15.865237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.865237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:15.865237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:15.865237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:15.865237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.865501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:15.865501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:15.865501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:15.865501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:15.865501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:15.865501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:15.865501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:15.865501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:15.865501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:15.865501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:15.865501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:15.865501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:15.865501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:15.865720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.865720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:15.865948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.865948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:15.865948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:15.865948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:15.865948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:15.865948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:15.865948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.865948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:15.865948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:15.865948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.865948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:15.865948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:15.865948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.865948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:15.865948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:15.865948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:15.866197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:15.866197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:15.866197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.866197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:15.866197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:15.866197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:15.866197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:15.866197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:15.866197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:15.866197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:15.866197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:15.866197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:15.866197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:15.866197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:15.866197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:15.866197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:15.866197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:15.866197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:15.866197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:15.866509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:15.866509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:15.866509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:18.734965 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:18.735435 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:18.735506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:18.736124 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:18.767725 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:18.767994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:18.768532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:18.768532 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:18.768532 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:18.768532 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:18.768532 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:18.768532 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:18.768532 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:18.768532 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:18.769270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:18.769270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:18.769270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:18.769270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:18.769270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:18.769270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:18.769270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:18.769270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:18.769270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:18.769270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:18.769270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:18.769270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:18.769493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:18.769493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:18.769493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:18.769493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:18.769493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:18.769493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:18.769493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:18.769493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:18.769493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:18.769493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:18.769493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:18.769493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:18.769493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:18.769690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:18.769690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:18.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:18.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:18.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:18.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:18.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:18.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:18.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:18.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:18.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:18.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:18.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:18.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:18.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:18.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:18.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:18.770010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:18.770275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:18.770275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:18.770275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:18.770275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:18.770275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:18.770275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:18.770275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:18.770275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:18.770275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:18.770275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:18.770275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:18.770275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:18.770275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:18.770275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:18.770275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:18.770275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:18.770275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:18.770275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:18.770275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:18.770562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:18.770562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:18.770562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:20.803232 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:20.803777 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:20.803777 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:20.804403 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:20.804557 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:20.804672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:20.804777 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e367330-4323-4718-94c3-86375e03fbd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:20.804913 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f424fd1-dda0-4816-aa5c-f5e45222260b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:20.805013 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:20.805112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:20.805203 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:20.805320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.805732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:20.805732 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:20.805732 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:20.805732 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:20.805732 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:20.805732 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:20.805732 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.805732 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:20.806372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.806372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:20.806372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:20.806372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:20.806372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.806372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:20.806372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:20.806372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.806372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:20.806372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:20.806372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:20.806372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.806579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:20.806579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:20.806579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:20.806579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:20.806579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:20.806579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:20.806579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:20.806579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:20.806579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:20.806579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:20.806579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:20.806579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:20.806579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:20.806771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.806771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:20.807047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.807047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:20.807047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:20.807047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:20.807047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:20.807047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:20.807047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.807047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:20.807047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:20.807047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.807047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:20.807047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:20.807047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.807047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:20.807047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:20.807047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.807326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:20.807326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:20.807326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.807326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:20.807326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:20.807326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:20.807326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.807326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:20.807326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:20.807326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:20.807326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:20.807326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:20.807326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:20.807326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:20.807326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:20.807326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:20.807326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:20.807326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:20.807326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:20.807628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:20.807628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.807628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:20.831722 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:20.831972 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:20.832082 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:20.832127 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:35:20.832435 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:20.832585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.832969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:20.832969 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:20.832969 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:20.832969 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:20.832969 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:20.832969 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:20.832969 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.832969 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:20.833498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.833498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:20.833498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:20.833498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:20.833498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.833498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:20.833498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:20.833498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:20.833498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.833498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:20.833498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:20.833498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:20.833498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:20.833498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:20.833498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:20.833768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:20.833768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:20.833768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:20.833768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:20.833768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:20.833768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:20.833768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:20.833768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.833768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:20.834061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.834061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:20.834061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:20.834061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:20.834061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.834061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:20.834061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:20.834061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:35:20.834061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:20.834061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:20.834061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.834061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:20.834290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:20.834290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:20.834290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.834290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:20.834290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:20.834290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:20.834290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:20.834290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:20.834290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:20.834290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:20.834290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:20.834290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:20.834290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:20.834290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:20.834524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:20.834524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:20.834524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.834524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:20.834728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.834728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:20.834728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:20.834728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:20.834728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:20.834728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:20.834728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.834728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:20.834728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:20.834728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.834728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:20.834728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:20.834728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.834728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:20.834728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:20.834728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.835495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:20.835495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:20.835495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:35:20.835495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:20.835495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:20.835495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.835495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:20.835495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:20.835495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:20.835495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.835495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:20.835495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:20.835495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:20.835495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:20.835495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:20.835495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:20.835495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:20.835495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:20.835852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:20.835852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:20.835852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:20.835852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:20.835852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:20.835852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.835852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:20.841699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:20.841966 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:20.842062 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:20.842160 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:20.842468 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:20.842643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.842987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:20.842987 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:20.842987 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:20.842987 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:20.842987 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:20.842987 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:20.842987 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.842987 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:20.843489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.843489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:20.843489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:20.843489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:20.843489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.843489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:20.843489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:20.843489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:20.843489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.843489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:20.843489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:20.843489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:20.843489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:20.843489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:20.843489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:20.843755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:20.843755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:20.843755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:20.843755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:20.843755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:20.843755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:20.843755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:20.843755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.843755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:20.844069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.844069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:20.844069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:20.844069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:20.844069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.844069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:20.844069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:20.844069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:20.844069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:20.844069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.844069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:20.844271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:20.844271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:20.844271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.844271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:20.844271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:20.844271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:20.844271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:20.844271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:20.844271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:20.844271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:20.844271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:20.844271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:20.844271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:20.844271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:20.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:20.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:20.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:20.844772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.844772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:20.844772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:20.844772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:20.844772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:20.844772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:20.844772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.844772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:20.844772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:20.844772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.844772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:20.844772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:20.844772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.844772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:20.844772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:20.844772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.844772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:20.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:20.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:20.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:20.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:20.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:20.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:20.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:20.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:20.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:20.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:20.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:20.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:20.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:20.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:20.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:20.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:20.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:20.845759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:20.845759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:20.845759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.845759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:20.863238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:20.863489 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:20.863590 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:20.863721 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:20.864005 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:20.864181 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:20.864391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:20.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:20.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:20.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:20.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:20.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:20.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:20.865366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.865366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:20.865366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:20.865366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:20.865366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.865366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:20.865366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:20.865366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.865366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:20.865366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:20.865366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:20.865366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.865564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:20.865564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:20.865564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:20.865564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:20.865564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:20.865564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:20.865564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:20.865564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:20.865564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:20.865564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:20.865564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:20.865564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:20.865564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:20.865769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.865769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:20.866005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.866005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:20.866005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:20.866005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:20.866005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:20.866005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:20.866005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.866005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:20.866005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:20.866005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.866005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:20.866005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:20.866005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.866005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:20.866005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:20.866005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:20.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:20.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:20.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:20.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:20.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:20.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:20.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:20.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:20.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:20.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:20.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:20.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:20.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:20.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:20.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:20.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:20.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:20.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:20.866980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:20.866980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:20.866980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:23.735367 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:23.735758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:23.735790 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:23.736503 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:23.775176 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:23.776426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:23.776984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:23.776984 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:23.776984 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:23.776984 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:23.776984 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:23.776984 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:23.776984 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:23.776984 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:23.778163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:23.778163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:23.778163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:23.778163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:23.778163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:23.778163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:23.778163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:23.778163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:23.778163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:23.778163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:23.778163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:23.778163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:23.778435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:23.778435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:23.778435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:23.778435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:23.778435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:23.778435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:23.778435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:23.778435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:23.778435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:23.778435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:23.778435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:23.778435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:23.778435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:23.778736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:23.778736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:23.778921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:23.778921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:23.778921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:23.778921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:23.778921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:23.778921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:23.778921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:23.778921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:23.778921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:23.778921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:23.778921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:23.778921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:23.778921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:23.778921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:23.778921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:23.778921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:23.779196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:23.779196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:23.779196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:23.779196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:23.779196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:23.779196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:23.779196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:23.779196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:23.779196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:23.779196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:23.779196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:23.779196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:23.779196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:23.779196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:23.779196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:23.779196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:23.779196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:23.779196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:23.779196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:23.779607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:23.779607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:23.779607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:25.803719 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:25.804224 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:25.804224 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:25.805338 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:25.805525 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:25.805692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:25.805859 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e367330-4323-4718-94c3-86375e03fbd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:25.806042 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:25.806193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:25.806381 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f424fd1-dda0-4816-aa5c-f5e45222260b, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:25.806528 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:25.806677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:25.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:25.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:25.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:25.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:25.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:25.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.807243 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:25.808226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.808226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:25.808226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:25.808226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:25.808226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.808226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:25.808226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:25.808226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.808226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:25.808226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:25.808226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:25.808226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.808472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:25.808472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:25.808472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:25.808472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:25.808472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:25.808472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:25.808472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:25.808472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:25.808472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:25.808472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:25.808472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:25.808472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:25.808472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:25.808688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.808688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:25.809413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.809413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:25.809413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:25.809413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:25.809413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:25.809413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:25.809413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.809413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:25.809413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:25.809413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.809413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:25.809413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:25.809413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.809413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:25.809413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:25.809413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.809695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:25.809695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:25.809695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.809695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:25.809695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:25.809695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:25.809695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.809695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:25.809695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:25.809695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:25.809695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:25.809695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:25.809695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:25.809695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:25.809695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:25.809695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:25.809695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:25.809695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:25.809695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:25.810026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:25.810026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.810026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:25.832184 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:25.832540 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:25.832648 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:25.832751 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:35:25.833081 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:25.833265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.833725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:25.833725 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:25.833725 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:25.833725 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:25.833725 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:25.833725 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:25.833725 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.833725 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:25.834302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.834302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:25.834302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:25.834302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:25.834302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.834302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:25.834302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:25.834302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:25.834302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.834302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:25.834302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:25.834302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:25.834302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:25.834302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:25.834302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:25.834633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:25.834633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:25.834633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:25.834633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:25.834633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:25.834633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:25.834633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:25.834633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.834633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:25.835113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.835113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:25.835113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:25.835113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:25.835113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.835113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:25.835113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:25.835113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:35:25.835113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:25.835113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:25.835113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.835113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:25.835368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:25.835368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:25.835368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.835368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:25.835368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:25.835368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:25.835368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:25.835368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:25.835368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:25.835368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:25.835368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:25.835368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:25.835368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:25.835368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:25.835629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:25.835629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:25.835629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.835629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:25.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:25.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:25.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:25.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:25.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:25.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:25.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:25.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:25.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:25.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:25.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:25.836020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.836331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:25.836331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:25.836331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:35:25.836331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:25.836331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:25.836331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.836331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:25.836331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:25.836331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:25.836331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.836331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:25.836331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:25.836331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:25.836331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:25.836331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:25.836331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:25.836331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:25.836331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:25.836687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:25.836687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:25.836687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:25.836687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:25.836687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:25.836687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.836687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:25.842022 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:25.842191 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:25.842305 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:25.842439 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:25.842678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:25.842855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.843189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:25.843189 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:25.843189 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:25.843189 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:25.843189 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:25.843189 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:25.843189 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.843189 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:25.843701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.843701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:25.843701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:25.843701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:25.843701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.843701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:25.843701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:25.843701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:25.843701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.843701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:25.843701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:25.843701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:25.843701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:25.843701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:25.843701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:25.844017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:25.844017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:25.844017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:25.844017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:25.844017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:25.844017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:25.844017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:25.844017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.844017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:25.844305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.844305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:25.844305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:25.844305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:25.844305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.844305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:25.844305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:25.844305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:25.844305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:25.844305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.844305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:25.844536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:25.844536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:25.844536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.844536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:25.844536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:25.844536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:25.844536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:25.844536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:25.844536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:25.844536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:25.844536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:25.844536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:25.844536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:25.844536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:25.844782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:25.844782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:25.844782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.844782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:25.845023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.845023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:25.845023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:25.845023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:25.845023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:25.845023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:25.845023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.845023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:25.845023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:25.845023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.845023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:25.845023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:25.845023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.845023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:25.845023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:25.845023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.845023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:25.845500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:25.845500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:25.845500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:25.845500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.845500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:25.845500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:25.845500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:25.845500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.845500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:25.845500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:25.845500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:25.845500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:25.845500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:25.845500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:25.845500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:25.845500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:25.845500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:25.845500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:25.845500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:25.845949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:25.845949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:25.845949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.845949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:25.863704 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:25.863983 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:25.864076 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:25.864213 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:25.865598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:25.865598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:25.865598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.865775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:25.865775 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:25.865775 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:25.865775 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:25.865775 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:25.865775 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:25.865775 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.865775 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:25.866734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.866734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:25.866734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:25.866734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:25.866734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.866734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:25.866734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:25.866734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.866734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:25.866734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:25.866734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:25.866734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.866978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:25.866978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:25.866978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:25.866978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:25.866978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:25.866978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:25.866978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:25.866978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:25.866978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:25.866978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:25.866978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:25.866978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:25.866978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:25.867189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.867189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:25.867900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.867900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:25.867900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:25.867900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:25.867900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:25.867900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:25.867900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.867900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:25.867900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:25.867900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.867900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:25.867900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:25.867900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.867900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:25.867900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:25.867900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:25.868180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:25.868180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:25.868180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.868180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:25.868180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:25.868180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:25.868180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:25.868180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:25.868180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:25.868180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:25.868180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:25.868180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:25.868180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:25.868180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:25.868180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:25.868180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:25.868180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:25.868180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:25.868180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:25.868507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:25.868507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:25.868507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:28.736228 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:28.736712 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:28.736712 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:28.737229 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:28.778438 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:28.778607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:28.779000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:28.779000 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:28.779000 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:28.779000 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:28.779000 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:28.779000 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:28.779000 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:28.779000 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:28.779648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:28.779648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:28.779648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:28.779648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:28.779648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:28.779648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:28.779648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:28.779648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:28.779648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:28.779648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:28.779648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:28.779648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:28.779931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:28.779931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:28.779931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:28.779931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:28.779931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:28.779931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:28.779931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:28.779931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:28.779931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:28.779931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:28.779931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:28.779931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:28.779931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:28.780135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:28.780135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:28.780611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:28.780611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:28.780611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:28.780611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:28.780611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:28.780611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:28.780611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:28.780611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:28.780611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:28.780611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:28.780611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:28.780611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:28.780611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:28.780611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:28.780611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:28.780611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:28.780901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:28.780901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:28.780901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:28.780901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:28.780901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:28.780901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:28.780901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:28.780901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:28.780901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:28.780901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:28.780901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:28.780901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:28.780901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:28.780901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:28.780901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:28.780901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:28.780901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:28.780901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:28.780901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:28.781182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:28.781182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:28.781182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:30.804219 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:30.805208 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:30.805330 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:30.806148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:30.806361 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:30.806482 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:30.806616 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e367330-4323-4718-94c3-86375e03fbd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:30.806716 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:30.806826 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:30.806972 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:30.807088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.807620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:30.807620 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:30.807620 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:30.807620 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:30.807620 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:30.807620 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:30.807620 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.807620 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:30.808399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.808399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:30.808399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:30.808399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:30.808399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.808399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:30.808399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:30.808399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.808399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:30.808399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:30.808399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:30.808399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.808684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:30.808684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:30.808684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:30.808684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:30.808684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:30.808684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:30.808684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:30.808684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:30.808684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:30.808684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:30.808684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:30.808684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:30.808684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:30.808982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.808982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:30.809564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.809564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:30.809564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:30.809564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:30.809564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:30.809564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:30.809564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.809564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:30.809564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:30.809564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.809564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:30.809564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:30.809564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.809564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:30.809564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:30.809564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.810526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:30.810526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:30.810526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.810526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:30.810526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:30.810526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:30.810526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.810526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:30.810526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:30.810526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:30.810526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:30.810526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:30.810526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:30.810526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:30.810526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:30.810526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:30.810526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:30.810526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:30.810526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:30.811485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:30.811485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.811485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:30.832677 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:30.833304 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:30.833304 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:30.833304 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:35:30.833762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:30.833934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.834536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:30.834536 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:30.834536 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:30.834536 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:30.834536 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:30.834536 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:30.834536 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.834536 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:30.835100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.835100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:30.835100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:30.835100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:30.835100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.835100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:30.835100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:30.835100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:30.835100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.835100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:30.835100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:30.835100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:30.835100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:30.835100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:30.835100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:30.835487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:30.835487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:30.835487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:30.835487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:30.835487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:30.835487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:30.835487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:30.835487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.835487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:30.835695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.835695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:30.835695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:30.835695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:30.835695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.835695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:30.835695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:30.835695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:35:30.835695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:30.835695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:30.835695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.835695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:30.835969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:30.835969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:30.835969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.835969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:30.835969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:30.835969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:30.835969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:30.835969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:30.835969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:30.835969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:30.835969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:30.835969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:30.835969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:30.835969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:30.836290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:30.836290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:30.836290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.836290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:30.836387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.836387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:30.836387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:30.836387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:30.836387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:30.836387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:30.836387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.836387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:30.836387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:30.836387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.836387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:30.836387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:30.836387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.836387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:30.836387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:30.836387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.836729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:30.836729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:30.836729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:35:30.836729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:30.836729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:30.836729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.836729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:30.836729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:30.836729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:30.836729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.836729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:30.836729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:30.836729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:30.836729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:30.836729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:30.836729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:30.836729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:30.836729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:30.837212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:30.837212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:30.837212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:30.837212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:30.837212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:30.837212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.837212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:30.842408 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:30.842603 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:30.842691 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:30.842795 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:30.843096 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:30.843293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.843644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:30.843644 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:30.843644 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:30.843644 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:30.843644 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:30.843644 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:30.843644 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.843644 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:30.844149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.844149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:30.844149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:30.844149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:30.844149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.844149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:30.844149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:30.844149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:30.844149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.844149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:30.844149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:30.844149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:30.844149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:30.844149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:30.844149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:30.844523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:30.844523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:30.844523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:30.844523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:30.844523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:30.844523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:30.844523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:30.844523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.844523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:30.844756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.844756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:30.844756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:30.844756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:30.844756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.844756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:30.844756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:30.844756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:30.844756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:30.844756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.844756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:30.845031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:30.845031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:30.845031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.845031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:30.845031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:30.845031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:30.845031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:30.845031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:30.845031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:30.845031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:30.845031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:30.845031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:30.845031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:30.845031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:30.845385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:30.845385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:30.845385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.845385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:30.845486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.845486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:30.845486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:30.845486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:30.845486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:30.845486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:30.845486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.845486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:30.845486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:30.845486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.845486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:30.845486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:30.845486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.845486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:30.845486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:30.845486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.845486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:30.845890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:30.845890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:30.845890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:30.845890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.845890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:30.845890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:30.845890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:30.845890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.845890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:30.845890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:30.845890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:30.845890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:30.845890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:30.845890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:30.845890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:30.845890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:30.845890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:30.845890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:30.845890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:30.846338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:30.846338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:30.846338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.846338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:30.864116 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:30.864341 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:30.864432 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:30.864572 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:30.865205 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:30.865205 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:30.865205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.865686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:30.865686 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:30.865686 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:30.865686 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:30.865686 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:30.865686 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:30.865686 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.865686 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:30.866135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.866135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:30.866135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:30.866135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:30.866135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.866135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:30.866135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:30.866135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.866135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:30.866135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:30.866135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:30.866135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.866540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:30.866540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:30.866540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:30.866540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:30.866540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:30.866540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:30.866540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:30.866540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:30.866540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:30.866540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:30.866540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:30.866540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:30.866540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:30.866852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.866852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:30.866852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.866852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:30.866852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:30.866852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:30.866852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:30.866852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:30.866852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.866852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:30.866852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:30.866852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.866852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:30.866852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:30.866852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.867224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:30.867224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:30.867224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:30.867224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:30.867224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:30.867224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.867224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:30.867224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:30.867224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:30.867224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:30.867224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:30.867224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:30.867224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:30.867224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:30.867224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:30.867224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:30.867224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:30.867224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:30.867224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:30.867688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:30.867688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:30.867688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:30.867688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:30.867688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:30.867688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:33.739561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:33.740029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:33.740029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:33.740694 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:33.780597 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:33.780870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:33.781557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:33.781557 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:33.781557 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:33.781557 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:33.781557 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:33.781557 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:33.781557 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:33.781557 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:33.783034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:33.783034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:33.783034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:33.783034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:33.783034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:33.783034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:33.783034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:33.783034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:33.783034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:33.783034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:33.783034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:33.783034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:33.783416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:33.783416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:33.783416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:33.783416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:33.783416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:33.783416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:33.783416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:33.783416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:33.783416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:33.783416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:33.783416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:33.783416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:33.783416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:33.783650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:33.783650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:33.783650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:33.783650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:33.783650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:33.783650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:33.783650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:33.783650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:33.783650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:33.783650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:33.783650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:33.783650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:33.783650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:33.783650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:33.783650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:33.784077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:33.784077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:33.784077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:33.784077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:33.784077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:33.784077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:33.784077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:33.784077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:33.784077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:33.784077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:33.784077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:33.784077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:33.784077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:33.784077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:33.784077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:33.784077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:33.784077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:33.784077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:33.784077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:33.784474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:33.784474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:33.784474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:33.784474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:33.784474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:33.784474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:35.805316 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:35.808625 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:35.809205 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:35.809425 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:35.809710 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:35.809775 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:35.809929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:35.810037 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:35.810133 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:35.810233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.810659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:35.810659 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:35.810659 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:35.810659 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:35.810659 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:35.810659 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:35.810659 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.810659 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:35.811299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.811299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:35.811299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:35.811299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:35.811299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.811299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:35.811299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:35.811299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.811299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:35.811299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:35.811299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:35.811299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.811679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:35.811679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:35.811679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:35.811679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:35.811679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:35.811679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:35.811679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:35.811679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:35.811679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:35.811679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:35.811679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:35.811679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:35.811679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:35.811929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.811929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:35.812733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.812733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:35.812733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:35.812733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:35.812733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:35.812733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:35.812733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.812733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:35.812733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:35.812733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.812733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:35.812733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:35.812733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.812733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:35.812733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:35.812733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.813507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:35.813507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:35.813507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.813507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:35.813507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:35.813507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:35.813507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.813507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:35.813507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:35.813507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:35.813507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:35.813507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:35.813507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:35.813507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:35.813507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:35.813507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:35.813507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:35.813507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:35.813507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:35.813989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:35.813989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.813989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:35.833100 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:35.833517 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:35.833517 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:35.833684 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:35:35.838285 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:35.838443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.838886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:35.838886 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:35.838886 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:35.838886 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:35.838886 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:35.838886 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:35.838886 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.838886 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:35.839567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.839567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:35.839567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:35.839567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:35.839567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.839567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:35.839567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:35.839567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:35.839567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.839567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:35.839567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:35.839567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:35.839567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:35.839567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:35.839567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:35.840181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:35.840181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:35.840181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:35.840181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:35.840181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:35.840181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:35.840181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:35.840181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.840181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:35.840181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.840181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:35.840181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:35.840181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:35.840181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.840526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:35.840526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:35.840526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:35:35.840526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:35.840526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:35.840526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.840526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:35.840526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:35.840526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:35.840526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.840526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:35.840526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:35.840526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:35.840843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:35.840843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:35.840843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:35.840843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:35.840843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:35.840843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:35.840843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:35.840843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:35.840843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:35.840843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:35.840843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.840843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:35.841136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.841136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:35.841136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:35.841136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:35.841136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:35.841136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:35.841136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.841136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:35.841136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:35.841136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.841136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:35.841136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:35.841136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.841136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:35.841136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:35.841136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.841457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:35.841457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:35.841457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:35:35.841457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:35.841457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:35.841457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.841457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:35.841457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:35.841457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:35.841457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.841457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:35.841457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:35.841457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:35.841457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:35.841457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:35.841457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:35.841457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:35.841457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:35.841800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:35.841800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:35.841800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:35.841800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:35.841800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:35.841800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.841800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:35.843648 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:35.843648 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:35.843648 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:35.843648 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:35.844450 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:35.844450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.845965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:35.845965 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:35.845965 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:35.845965 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:35.845965 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:35.845965 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:35.845965 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.845965 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:35.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:35.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:35.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:35.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:35.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:35.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:35.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:35.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:35.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:35.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:35.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:35.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:35.849203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:35.849203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:35.849203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:35.849203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:35.849203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:35.849203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:35.849203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:35.849203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.849203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:35.849203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.849203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:35.849203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:35.849203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:35.849203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.849495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:35.849495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:35.849495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:35.849495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:35.849495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.849495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:35.849495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:35.849495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:35.849495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.849495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:35.849495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:35.849495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:35.849495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:35.849725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:35.849725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:35.849725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:35.849725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:35.849725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:35.849725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:35.849725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:35.849725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:35.849725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:35.849725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.849725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:35.849725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.849967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:35.849967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:35.849967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:35.849967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:35.849967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:35.849967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.849967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:35.849967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:35.849967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.849967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:35.849967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:35.849967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.849967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:35.849967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:35.849967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.849967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:35.849967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:35.849967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:35.849967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.850318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:35.864492 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:35.865388 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:35.865533 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:35.865687 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:35.865970 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:35.866228 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:35.866817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.869478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:35.869478 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:35.869478 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:35.869478 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:35.869478 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:35.869478 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:35.869478 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.869478 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:35.870375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.870375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:35.870375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:35.870375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:35.870375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.870375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:35.870375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:35.870375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.870375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:35.870375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:35.870375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:35.870375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.870614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:35.870614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:35.870614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:35.870614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:35.870614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:35.870614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:35.870614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:35.870614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:35.870614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:35.870614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:35.870614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:35.870614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:35.870614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:35.870815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.870815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:35.871048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.871048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:35.871048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:35.871048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:35.871048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:35.871048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:35.871048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.871048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:35.871048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:35.871048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.871048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:35.871048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:35.871048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.871048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:35.871048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:35.871048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:35.871374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:35.871374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:35.871374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.871374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:35.871374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:35.871374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:35.871374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:35.871374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:35.871374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:35.871374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:35.871374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:35.871374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:35.871374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:35.871374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:35.871374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:35.871374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:35.871374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:35.871374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:35.871374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:35.871668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:35.871668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:35.871668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:38.740039 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:38.740478 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:38.740580 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:38.741166 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:38.774525 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:38.774701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:38.775151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:38.775151 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:38.775151 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:38.775151 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:38.775151 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:38.775151 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:38.775151 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:38.775151 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:38.775770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:38.775770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:38.775770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:38.775770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:38.775770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:38.775770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:38.775770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:38.775770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:38.775770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:38.775770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:38.775770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:38.775770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:38.776058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:38.776058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:38.776058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:38.776058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:38.776058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:38.776058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:38.776058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:38.776058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:38.776058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:38.776058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:38.776058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:38.776058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:38.776058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:38.776343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:38.776343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:38.776717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:38.776717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:38.776717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:38.776717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:38.776717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:38.776717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:38.776717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:38.776717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:38.776717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:38.776717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:38.776717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:38.776717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:38.776717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:38.776717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:38.776717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:38.776717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:38.777011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:38.777011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:38.777011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:38.777011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:38.777011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:38.777011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:38.777011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:38.777011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:38.777011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:38.777011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:38.777011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:38.777011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:38.777011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:38.777011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:38.777011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:38.777011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:38.777011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:38.777011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:38.777011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:38.777329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:38.777329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:38.777329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:40.820522 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:40.820845 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:40.820845 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:40.825230 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:40.825473 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:40.825656 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:40.825825 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:40.825991 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:40.826205 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:40.826379 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:40.826523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:40.826643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.829847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:40.829847 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:40.829847 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:40.829847 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:40.829847 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:40.829847 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:40.829847 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.829847 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:40.830905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.830905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:40.830905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:40.830905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:40.830905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.830905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:40.830905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:40.830905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.830905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:40.830905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:40.830905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:40.830905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.831126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:40.831126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:40.831126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:40.831126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:40.831126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:40.831126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:40.831126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:40.831126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:40.831126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:40.831126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:40.831126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:40.831126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:40.831126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:40.831345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.831345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:40.832781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.832781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:40.832781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:40.832781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:40.832781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:40.832781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:40.832781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.832781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:40.832781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:40.832781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.832781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:40.832781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:40.832781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.832781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:40.832781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:40.832781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.833244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:40.833244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:40.833244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.833244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:40.833244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:40.833244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:40.833244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.833244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:40.833244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:40.833244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:40.833244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:40.833244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:40.833244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:40.833244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:40.833244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:40.833244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:40.833244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:40.833244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:40.833244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:40.833840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:40.833840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.833840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:40.833975 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:40.834183 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:40.834282 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:40.834378 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:35:40.834986 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:40.835142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.835709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:40.835709 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:40.835709 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:40.835709 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:40.835709 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:40.835709 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:40.835709 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.835709 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:40.836221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.836221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:40.836221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:40.836221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:40.836221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.836221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:40.836221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:40.836221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:40.836221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.836221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:40.836221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:40.836221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:40.836221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:40.836221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:40.836221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:40.836668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:40.836668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:40.836668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:40.836668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:40.836668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:40.836668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:40.836668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:40.836668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.836668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:40.836994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.836994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:40.836994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:40.836994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:40.836994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.836994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:40.836994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:40.836994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:35:40.836994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:40.836994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:40.836994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.836994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:40.837478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:40.837478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:40.837478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.837478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:40.837478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:40.837478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:40.837478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:40.837478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:40.837478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:40.837478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:40.837478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:40.837478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:40.837478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:40.837478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:40.837930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:40.837930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:40.837930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.837930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:40.837930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.837930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:40.837930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:40.837930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:40.837930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:40.837930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:40.837930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.837930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:40.837930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:40.837930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.838346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:40.838346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:40.838346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.838346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:40.838346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:40.838346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.838346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:40.838346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:40.838346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:35:40.838346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:40.838346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:40.838346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.838346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:40.838346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:40.838346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:40.838346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.838346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:40.838346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:40.838346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:40.838869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:40.838869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:40.838869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:40.838869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:40.838869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:40.838869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:40.838869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:40.838869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:40.838869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:40.838869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:40.838869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.838869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:40.843172 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:40.843584 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:40.843725 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:40.843830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:40.844153 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:40.844408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.844794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:40.844794 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:40.844794 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:40.844794 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:40.844794 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:40.844794 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:40.844794 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.844794 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:40.845373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.845373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:40.845373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:40.845373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:40.845373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.845373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:40.845373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:40.845373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:40.845373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.845373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:40.845373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:40.845373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:40.845373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:40.845373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:40.845373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:40.845611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:40.845611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:40.845611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:40.845611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:40.845611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:40.845611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:40.845611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:40.845611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.845611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:40.846472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.846472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:40.846472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:40.846472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:40.846472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.846472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:40.846472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:40.846472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:40.846472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:40.846472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.846472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:40.847012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:40.847012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:40.847012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.847012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:40.847012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:40.847012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:40.847012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:40.847012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:40.847012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:40.847012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:40.847012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:40.847012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:40.847012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:40.847012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:40.847276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:40.847276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:40.847276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.847276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:40.847276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.847276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:40.847276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:40.847276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:40.847276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:40.847276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:40.847276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.847276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:40.847276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:40.847276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.847597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:40.847597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:40.847597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.847597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:40.847597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:40.847597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.847597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:40.847597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:40.847597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:40.847597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:40.847597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.847597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:40.847597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:40.847597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:40.847597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.847597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:40.847597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:40.847597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:40.847597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:40.848013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:40.848013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:40.848013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:40.848013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:40.848013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:40.848013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:40.848013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:40.848013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:40.848013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:40.848013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.848013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:40.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:40.865884 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:40.866011 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:40.866168 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:40.866467 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:40.866672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:40.866918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.867334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:40.867334 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:40.867334 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:40.867334 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:40.867334 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:40.867334 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:40.867334 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.867334 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:40.867970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.867970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:40.867970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:40.867970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:40.867970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.867970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:40.867970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:40.867970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.867970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:40.867970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:40.867970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:40.867970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.868384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:40.868384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:40.868384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:40.868384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:40.868384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:40.868384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:40.868384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:40.868384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:40.868384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:40.868384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:40.868384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:40.868384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:40.868384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:40.868787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.868787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:40.868787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.868787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:40.868787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:40.868787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:40.868787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:40.868787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:40.868787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.868787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:40.868787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:40.868787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.868787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:40.868787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:40.868787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.869260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:40.869260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:40.869260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:40.869260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:40.869260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:40.869260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.869260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:40.869260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:40.869260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:40.869260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:40.869260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:40.869260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:40.869260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:40.869260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:40.869260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:40.869260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:40.869260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:40.869260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:40.869260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:40.869831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:40.869831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:40.869831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:40.869831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:40.869831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:40.869831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:43.740587 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:43.741204 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:43.741204 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:43.741985 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:43.776410 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:43.776684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:43.777360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:43.777360 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:43.777360 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:43.777360 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:43.777360 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:43.777360 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:43.777360 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:43.777360 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:43.778323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:43.778323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:43.778323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:43.778323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:43.778323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:43.778323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:43.778323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:43.778323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:43.778323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:43.778323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:43.778323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:43.778323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:43.778578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:43.778578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:43.778578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:43.778578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:43.778578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:43.778578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:43.778578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:43.778578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:43.778578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:43.778578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:43.778578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:43.778578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:43.778578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:43.778817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:43.778817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:43.779123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:43.779123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:43.779123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:43.779123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:43.779123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:43.779123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:43.779123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:43.779123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:43.779123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:43.779123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:43.779123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:43.779123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:43.779123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:43.779123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:43.779123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:43.779123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:43.779512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:43.779512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:43.779512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:43.779512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:43.779512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:43.779512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:43.779512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:43.779512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:43.779512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:43.779512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:43.779512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:43.779512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:43.779512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:43.779512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:43.779512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:43.779512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:43.779512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:43.779512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:43.779512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:43.779836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:43.779836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:43.779836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:45.820756 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:45.821192 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:45.821192 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:45.822541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:45.822733 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:45.822873 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:45.822982 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:45.823082 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:45.823181 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:45.823317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:45.823421 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:45.823524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.823996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:45.823996 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:45.823996 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:45.823996 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:45.823996 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:45.823996 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:45.823996 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.823996 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:45.824659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.824659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:45.824659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:45.824659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:45.824659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.824659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:45.824659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:45.824659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.824659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:45.824659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:45.824659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:45.824659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.824906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:45.824906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:45.824906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:45.824906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:45.824906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:45.824906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:45.824906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:45.824906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:45.824906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:45.824906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:45.824906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:45.824906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:45.824906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:45.825107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.825107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:45.825363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.825363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:45.825363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:45.825363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:45.825363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:45.825363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:45.825363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.825363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:45.825363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:45.825363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.825363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:45.825363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:45.825363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.825363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:45.825363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:45.825363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.825631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:45.825631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:45.825631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.825631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:45.825631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:45.825631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:45.825631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.825631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:45.825631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:45.825631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:45.825631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:45.825631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:45.825631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:45.825631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:45.825631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:45.825631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:45.825631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:45.825631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:45.825631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:45.825952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:45.825952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.825952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:45.834295 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:45.834463 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:45.834541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:45.834636 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:35:45.835397 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:45.835547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.836000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:45.836000 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:45.836000 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:45.836000 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:45.836000 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:45.836000 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:45.836000 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.836000 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:45.836861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.836861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:45.836861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:45.836861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:45.836861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.836861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:45.836861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:45.836861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:45.836861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.836861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:45.836861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:45.836861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:45.836861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:45.836861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:45.836861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:45.837120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:45.837120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:45.837120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:45.837120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:45.837120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:45.837120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:45.837120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:45.837120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.837120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:45.837765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.837765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:45.837765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:45.837765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:45.837765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.837765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:45.837765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:45.837765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:35:45.837765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:45.837765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:45.837765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.837765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:45.837987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:45.837987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:45.837987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.837987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:45.837987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:45.837987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:45.837987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:45.837987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:45.837987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:45.837987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:45.837987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:45.837987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:45.837987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:45.837987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:45.838193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:45.838193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:45.838193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.838193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:45.838930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.838930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:45.838930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:45.838930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:45.838930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:45.838930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:45.838930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.838930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:45.838930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:45.838930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.838930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:45.838930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:45.838930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.838930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:45.838930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:45.838930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.839198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:45.839198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:45.839198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:35:45.839198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:45.839198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:45.839198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.839198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:45.839198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:45.839198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:45.839198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.839198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:45.839198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:45.839198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:45.839198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:45.839198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:45.839198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:45.839198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:45.839198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:45.839496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:45.839496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:45.839496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:45.839496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:45.839496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:45.839496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.839496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:45.843644 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:45.843949 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:45.844077 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:45.844220 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:45.844536 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:45.844730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.845192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:45.845192 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:45.845192 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:45.845192 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:45.845192 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:45.845192 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:45.845192 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.845192 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:45.846061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.846061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:45.846061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:45.846061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:45.846061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.846061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:45.846061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:45.846061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:45.846061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.846061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:45.846061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:45.846061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:45.846061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:45.846061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:45.846061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:45.846334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:45.846334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:45.846334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:45.846334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:45.846334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:45.846334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:45.846334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:45.846334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.846334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:45.847061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.847061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:45.847061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:45.847061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:45.847061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.847061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:45.847061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:45.847061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:45.847061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:45.847061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.847061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:45.847327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:45.847327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:45.847327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.847327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:45.847327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:45.847327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:45.847327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:45.847327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:45.847327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:45.847327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:45.847327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:45.847327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:45.847327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:45.847327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:45.847572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:45.847572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:45.847572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.847572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:45.848192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.848192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:45.848192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:45.848192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:45.848192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:45.848192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:45.848192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.848192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:45.848192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:45.848192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.848192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:45.848192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:45.848192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.848192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:45.848192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:45.848192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.848192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:45.848593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:45.848593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:45.848593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:45.848593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.848593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:45.848593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:45.848593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:45.848593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.848593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:45.848593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:45.848593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:45.848593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:45.848593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:45.848593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:45.848593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:45.848593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:45.848593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:45.848593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:45.848593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:45.848973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:45.848973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:45.848973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.848973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:45.865869 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:45.866121 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:45.866209 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:45.866391 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:45.867581 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:45.867783 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:45.868012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.868452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:45.868452 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:45.868452 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:45.868452 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:45.868452 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:45.868452 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:45.868452 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.868452 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:45.869174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.869174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:45.869174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:45.869174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:45.869174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.869174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:45.869174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:45.869174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.869174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:45.869174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:45.869174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:45.869174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.869466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:45.869466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:45.869466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:45.869466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:45.869466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:45.869466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:45.869466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:45.869466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:45.869466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:45.869466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:45.869466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:45.869466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:45.869466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:45.869680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.869680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:45.870082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.870082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:45.870082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:45.870082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:45.870082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:45.870082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:45.870082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.870082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:45.870082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:45.870082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.870082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:45.870082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:45.870082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.870082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:45.870082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:45.870082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:45.870402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:45.870402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:45.870402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.870402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:45.870402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:45.870402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:45.870402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:45.870402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:45.870402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:45.870402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:45.870402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:45.870402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:45.870402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:45.870402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:45.870402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:45.870402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:45.870402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:45.870402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:45.870402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:45.870824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:45.870824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:45.870824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:48.741037 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:48.741479 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:48.741529 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:48.742284 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:48.782401 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:48.782651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:48.783234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:48.783234 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:48.783234 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:48.783234 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:48.783234 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:48.783234 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:48.783234 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:48.783234 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:48.784454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:48.784454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:48.784454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:48.784454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:48.784454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:48.784454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:48.784454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:48.784454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:48.784454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:48.784454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:48.784454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:48.784454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:48.784717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:48.784717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:48.784717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:48.784717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:48.784717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:48.784717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:48.784717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:48.784717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:48.784717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:48.784717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:48.784717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:48.784717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:48.784717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:48.784947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:48.784947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:48.785726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:48.785726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:48.785726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:48.785726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:48.785726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:48.785726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:48.785726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:48.785726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:48.785726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:48.785726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:48.785726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:48.785726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:48.785726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:48.785726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:48.785726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:48.785726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:48.786038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:48.786038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:48.786038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:48.786038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:48.786038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:48.786038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:48.786038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:48.786038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:48.786038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:48.786038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:48.786038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:48.786038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:48.786038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:48.786038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:48.786038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:48.786038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:48.786038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:48.786038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:48.786038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:48.786421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:48.786421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:48.786421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:50.821130 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:50.822755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:50.822755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:50.823511 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:50.823511 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:50.823511 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:50.823511 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:50.823511 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:50.823742 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:50.823742 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-719622d3-2439-47ea-ae1a-087c9f0d1b20, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:50.823742 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a1f0ea6-0db5-4f0a-82d2-3c73ac180ba4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:50.823898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.824790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:50.824790 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:50.824790 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:50.824790 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:50.824790 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:50.824790 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:50.824790 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.824790 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:50.826782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.826782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:50.826782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:50.826782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:50.826782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.826782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:50.826782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:50.826782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.826782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:50.826782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:50.826782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:50.826782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.827196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:50.827196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:50.827196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:50.827196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:50.827196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:50.827196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:50.827196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:50.827196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:50.827196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:50.827196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:50.827196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:50.827196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:50.827196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:50.827569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.827569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:50.829746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.829746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:50.829746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:50.829746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:50.829746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:50.829746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:50.829746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.829746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:50.829746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:50.829746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.829746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:50.829746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:50.829746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.829746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:50.829746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:50.829746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.830276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:50.830276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:50.830276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.830276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:50.830276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:50.830276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:50.830276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.830276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:50.830276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:50.830276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:50.830276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:50.830276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:50.830276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:50.830276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:50.830276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:50.830276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:50.830276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:50.830276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:50.830276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:50.830806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:50.830806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.830806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:50.835526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:50.835526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:50.835526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:50.835526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:35:50.835526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:50.835766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.836396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:50.836396 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:50.836396 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:50.836396 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:50.836396 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:50.836396 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:50.836396 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.836396 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:50.839646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.839646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:50.839646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:50.839646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:50.839646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.839646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:50.839646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:50.839646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:50.839646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.839646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:50.839646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:50.839646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:50.839646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:50.839646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:50.839646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:50.840136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:50.840136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:50.840136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:50.840136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:50.840136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:50.840136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:50.840136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:50.840136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.840136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:50.840136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.840136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:50.840136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:50.840136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:50.840136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.840545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:50.840545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:50.840545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:35:50.840545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:50.840545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:50.840545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.840545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:50.840545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:50.840545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:50.840545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.840545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:50.840545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:50.840545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:50.840919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:50.840919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:50.840919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:50.840919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:50.840919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:50.840919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:50.840919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:50.840919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:50.840919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:50.840919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:50.840919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.840919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:50.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:50.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:50.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:50.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:50.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:50.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:50.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:50.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:50.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:50.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:50.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:50.841269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.841782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:50.841782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:50.841782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:35:50.841782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:50.841782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:50.841782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.841782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:50.841782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:50.841782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:50.841782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.841782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:50.841782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:50.841782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:50.841782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:50.841782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:50.841782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:50.841782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:50.841782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:50.842267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:50.842267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:50.842267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:50.842267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:50.842267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:50.842267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.842267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:50.843983 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:50.844667 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:50.844667 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:50.844667 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:50.844986 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:50.845186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.846283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:50.846283 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:50.846283 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:50.846283 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:50.846283 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:50.846283 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:50.846283 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.846283 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:50.846283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.846283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:50.846283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:50.846283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:50.846283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.846283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:50.846283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:50.846283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:50.846283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.846283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:50.846787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:50.846787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:50.846787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:50.846787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:50.846787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:50.846787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:50.846787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:50.846787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:50.846787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:50.846787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:50.846787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:50.846787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:50.846787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.846787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:50.848363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.848363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:50.848363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:50.848363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:50.848363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.848363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:50.848363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:50.848363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:50.848363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:50.848363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.848363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:50.848715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:50.848715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:50.848715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.848715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:50.848715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:50.848715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:50.848715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:50.848715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:50.848715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:50.848715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:50.848715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:50.848715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:50.848715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:50.848715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:50.849110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:50.849110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:50.849110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.849110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:50.850031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.850031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:50.850031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:50.850031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:50.850031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:50.850031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:50.850031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.850031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:50.850031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:50.850031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.850031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:50.850031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:50.850031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.850031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:50.850031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:50.850031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.850031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:50.850543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:50.850543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:50.850543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:50.850543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.850543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:50.850543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:50.850543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:50.850543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.850543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:50.850543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:50.850543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:50.850543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:50.850543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:50.850543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:50.850543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:50.850543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:50.850543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:50.850543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:50.850543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:50.851041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:50.851041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:50.851041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.851041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:50.866266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:50.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:50.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:50.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:50.867367 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:50.867614 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:50.867890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.868359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:50.868359 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:50.868359 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:50.868359 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:50.868359 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:50.868359 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:50.868359 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.868359 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:50.870115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.870115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:50.870115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:50.870115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:50.870115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.870115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:50.870115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:50.870115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.870115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:50.870115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:50.870115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:50.870115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.870493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:50.870493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:50.870493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:50.870493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:50.870493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:50.870493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:50.870493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:50.870493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:50.870493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:50.870493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:50.870493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:50.870493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:50.870493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:50.870864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.870864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:50.870864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.870864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:50.870864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:50.870864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:50.870864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:50.870864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:50.870864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.870864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:50.870864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:50.870864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.870864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:50.870864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:50.870864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.871265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:50.871265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:50.871265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:50.871265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:50.871265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:50.871265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.871265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:50.871265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:50.871265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:50.871265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:50.871265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:50.871265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:50.871265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:50.871265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:50.871265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:50.871265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:50.871265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:50.871265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:50.871265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:50.871769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:50.871769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:50.871769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:50.871769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:50.871769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:50.871769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:53.741500 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:53.742599 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:53.742707 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:53.743320 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:53.751352 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=240 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:35:53.787627 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:53.788184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:53.788850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:53.788850 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:53.788850 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:53.788850 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:53.788850 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:53.788850 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:53.788850 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:53.788850 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:53.789571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:53.789571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:53.789571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:53.789571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:53.789571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:53.789571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:53.789571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:53.789571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:53.789571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:53.789571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:53.789571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:53.789571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:53.789835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:53.789835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:53.789835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:53.789835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:53.789835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:53.789835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:53.789835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:53.789835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:53.789835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:53.789835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:53.789835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:53.789835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:53.789835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:53.790079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:53.790079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:53.790313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:53.790313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:53.790313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:53.790313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:53.790313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:53.790313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:53.790313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:53.790313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:53.790313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:53.790313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:53.790313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:53.790313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:53.790313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:53.790313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:53.790313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:53.790313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:53.790701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:53.790701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:53.790701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:53.790701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:53.790701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:53.790701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:53.790701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:53.790701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:53.790701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:53.790701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:53.790701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:53.790701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:53.790701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:53.790701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:53.790701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:53.790701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:53.790701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:53.790701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:53.790701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:53.791086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:53.791086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:53.791086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:55.823278 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:55.824025 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:55.824168 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:55.826349 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:55.826607 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-63580fb6-c257-475a-8110-248c458342b1, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:55.826810 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:55.826973 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:55.827109 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:55.827267 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:55.827424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.828069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:55.828069 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:55.828069 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:55.828069 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:55.828069 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:55.828069 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:55.828069 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.828069 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:55.829480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.829480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:55.829480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:55.829480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:55.829480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.829480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:55.829480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:55.829480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.829480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:55.829480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:55.829480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:55.829480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.829841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:55.829841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:55.829841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:55.829841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:55.829841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:55.829841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:55.829841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:55.829841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:55.829841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:55.829841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:55.829841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:55.829841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:55.829841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:55.830116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.830116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:55.833365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.833365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:55.833365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:55.833365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:55.833365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:55.833365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:55.833365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.833365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:55.833365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:55.833365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.833365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:55.833365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:55.833365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.833365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:55.833365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:55.833365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.833690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:35:55.833690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:55.833690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.833690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:55.833690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:55.833690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:55.833690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.833690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:55.833690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:55.833690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:55.833690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:55.833690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:55.833690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:55.833690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:55.833690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:55.833690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:55.833690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:55.833690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:55.833690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:55.836523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:55.836523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.836523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:55.836523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:55.836523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:55.836523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:55.836523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:35:55.836897 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:55.837112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.838685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:55.838685 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:55.838685 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:55.838685 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:55.838685 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:55.838685 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:55.838685 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.838685 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:55.839328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.839328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:55.839328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:55.839328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:55.839328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.839328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:55.839328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:55.839328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:55.839328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.839328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:55.839328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:55.839328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:55.839328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:55.839328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:55.839328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:55.839616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:55.839616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:55.839616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:55.839616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:55.839616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:55.839616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:55.839616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:55.839616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.839616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:55.840531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.840531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:55.840531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:55.840531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:55.840531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.840531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:55.840531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:55.840531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:35:55.840531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:55.840531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:55.840531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.840531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:55.840907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:55.840907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:55.840907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.840907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:55.840907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:55.840907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:55.840907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:55.840907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:55.840907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:55.840907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:55.840907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:55.840907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:55.840907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:55.840907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:55.841121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:55.841121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:55.841121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.841121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:55.841880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.841880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:55.841880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:55.841880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:55.841880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:55.841880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:55.841880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.841880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:55.841880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:55.841880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.841880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:55.841880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:55.841880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.841880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:55.841880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:55.841880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.842186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:35:55.842186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:35:55.842186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:35:55.842186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:55.842186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:55.842186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.842186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:55.842186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:55.842186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:55.842186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.842186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:55.842186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:55.842186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:55.842186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:55.842186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:55.842186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:55.842186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:55.842186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:55.842539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:55.842539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:55.842539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:55.842539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:55.842539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:55.842539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.842539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:55.844330 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:55.844619 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:55.844750 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:55.844914 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:55.845419 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:55.845675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.848181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:55.848181 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:55.848181 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:55.848181 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:55.848181 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:55.848181 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:55.848181 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.848181 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:55.848802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.848802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:35:55.848802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:55.848802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:55.848802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.848802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:55.848802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:55.848802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:35:55.848802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.848802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:35:55.848802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:55.848802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:35:55.848802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:55.848802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:35:55.848802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:35:55.849152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:55.849152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:35:55.849152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:55.849152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:35:55.849152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:35:55.849152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:55.849152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:35:55.849152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.849152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:35:55.850393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.850393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:55.850393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:55.850393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:55.850393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.850393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:55.850393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:55.850393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:55.850393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:55.850393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.850393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:55.850658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:55.850658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:55.850658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.850658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:55.850658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:55.850658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:55.850658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:55.850658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:55.850658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:55.850658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:55.850658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:55.850658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:55.850658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:55.850658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:55.850902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:55.850902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:55.850902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.850902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:55.852335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.852335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:55.852335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:55.852335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:55.852335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:55.852335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:55.852335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.852335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:55.852335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:55.852335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.852335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:55.852335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:55.852335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.852335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:55.852335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:55.852335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.852335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:35:55.852660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:35:55.852660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:35:55.852660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:35:55.852660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.852660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:55.852660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:55.852660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:55.852660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.852660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:55.852660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:55.852660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:55.852660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:55.852660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:55.852660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:55.852660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:55.852660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:55.852660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:55.852660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:55.852660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:55.852992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:55.852992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:55.852992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.852992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:55.869534 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:55.869933 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:55.870075 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:55.870351 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:55.871987 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:55.872269 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:55.874692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.875362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:55.875362 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:55.875362 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:55.875362 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:55.875362 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:55.875362 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:55.875362 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.875362 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:55.875954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.875954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:55.875954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:55.875954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:55.875954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.875954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:55.875954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:55.875954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.875954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:55.875954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:55.875954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:55.875954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.876285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:55.876285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:55.876285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:55.876285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:55.876285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:55.876285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:55.876285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:55.876285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:55.876285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:55.876285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:55.876285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:55.876285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:55.876285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:55.876512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.876512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:55.879272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.879272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:55.879272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:55.879272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:55.879272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:55.879272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:55.879272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.879272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:55.879272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:55.879272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.879272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:55.879272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:55.879272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.879272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:55.879272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:55.879272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:55.880067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:35:55.880067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:55.880067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.880067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:55.880067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:55.880067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:55.880067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:55.880067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:55.880067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:55.880067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:55.880067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:55.880067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:55.880067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:55.880067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:55.880067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:55.880067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:55.880067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:55.880067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:55.880067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:55.880435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:55.880435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:55.880435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:58.742668 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:58.743526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:58.743618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:58.744201 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:35:58.790140 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:58.790140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:58.790355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:58.790355 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:58.790355 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:58.790355 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:58.790355 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:58.790355 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:58.790355 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:58.790355 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:58.791520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:58.791520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:58.791520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:58.791520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:35:58.791520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:58.791520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:58.791520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:58.791520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:58.791520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:58.791520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:58.791520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:58.791520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:58.791824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:58.791824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:58.791824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:58.791824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:58.791824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:58.791824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:58.791824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:58.791824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:58.791824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:58.791824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:58.791824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:58.791824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:58.791824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:58.792074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:58.792074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:58.792881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:58.792881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:35:58.792881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:35:58.792881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:35:58.792881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:35:58.792881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:35:58.792881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:58.792881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:35:58.792881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:35:58.792881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:58.792881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:35:58.792881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:35:58.792881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:35:58.792881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:35:58.792881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:35:58.792881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:35:58.793215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:35:58.793215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:58.793215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:58.793215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:58.793215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:58.793215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:35:58.793215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:58.793215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:35:58.793215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:58.793215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:35:58.793215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:58.793215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:35:58.793215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:35:58.793215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:58.793215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:35:58.793215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:58.793215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:35:58.793215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:35:58.793215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:58.793570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:35:58.793570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:58.793570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:35:58.859640 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:35:58.861375 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:35:58.861375 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:35:58.875007 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 08 14:35:58.875216 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Number of inconsistencies found at create/update: networks=5, routers=2, router_ports=2 {{(pid=89274) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 14:35:58.875407 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=16 {{(pid=89274) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 14:35:58.875550 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource a137566a-eaf1-4f5d-a192-15d63e6c049a (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:35:58.916896 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-a137566a-eaf1-4f5d-a192-15d63e6c049a, resource={'id': 'a137566a-eaf1-4f5d-a192-15d63e6c049a', 'name': 'private', 'tenant_id': 'ba473288446843d3a29e25267968ba61', 'project_id': 'ba473288446843d3a29e25267968ba61', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['7b3cb50f-b942-4054-b735-5f7be989b8f8', 'ba10f097-6c2b-4188-8d42-e9415eff0039'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:20:42Z', 'updated_at': '2026-06-08T14:20:44Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 26551}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:58.916896 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-a137566a-eaf1-4f5d-a192-15d63e6c049a, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:58.917059 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-c4b70b51-9d46-4e6b-a1e7-b3e6a6b8b666, external_ids_update=None, columns={'tag': 26551}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:58.917059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:58.917138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:58.917138 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:58.917138 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:58.917138 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:58.917138 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:58.917138 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:58.917138 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:58.917138 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:58.918035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource a137566a-eaf1-4f5d-a192-15d63e6c049a (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:58.918035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:58.918035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:35:58.918035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:35:58.918035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:35:58.918035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:35:58.918035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:35:58.918035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:58.918035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:58.918035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:58.918035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:58.918361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:58.918361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:58.918361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:58.918361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:58.918361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:58.918361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:58.918361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:58.918361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:58.918361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:58.918361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:58.918361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:58.918361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:58.918361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:58.918361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:58.918621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:58.918621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:58.918621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:58.918621 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource fffd4e9d-b343-4a97-8e5f-4015f805fc46 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:35:58.944597 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, resource={'id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'name': 'public', 'tenant_id': 'ba473288446843d3a29e25267968ba61', 'project_id': 'ba473288446843d3a29e25267968ba61', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['4ee83802-2181-432f-a76d-d3335bbf4e0b', 'b9859e49-8026-492d-a0a4-11326477713e'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-08T14:20:46Z', 'updated_at': '2026-06-08T14:20:52Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:58.946313 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:58.946313 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-0817ce61-1c85-4457-911a-0b8fdccf389d, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:58.946313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:58.946462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:58.946462 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:58.946462 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:58.946462 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:58.946462 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:58.946462 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:58.946462 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:58.946462 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:58.947740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource fffd4e9d-b343-4a97-8e5f-4015f805fc46 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:58.947740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:58.947740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:35:58.947740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:35:58.947740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:35:58.947740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:35:58.947740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:35:58.947740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:58.947740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:58.947740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:58.947740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:58.948022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:58.948022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:58.948022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:58.948022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:58.948022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:58.948022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:58.948022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:58.948022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:58.948022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:58.948022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:58.948022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:58.948022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:58.948022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:58.948022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:58.948287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:58.948287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:58.948287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:58.948287 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 73426422-a7fc-44cd-9fc5-cd2d8fa95c38 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:35:58.978646 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-73426422-a7fc-44cd-9fc5-cd2d8fa95c38, resource={'id': '73426422-a7fc-44cd-9fc5-cd2d8fa95c38', 'name': 'shared', 'tenant_id': 'ba473288446843d3a29e25267968ba61', 'project_id': 'ba473288446843d3a29e25267968ba61', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['2b53474a-74e5-4c44-bb1b-86c8215bd7ab'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:21:52Z', 'updated_at': '2026-06-08T14:21:52Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 292}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:58.978646 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-73426422-a7fc-44cd-9fc5-cd2d8fa95c38, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:58.978807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-5c6a587e-b2f4-42c7-9706-8a2e20b0df46, external_ids_update=None, columns={'tag': 292}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:58.978807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:58.978807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:58.978807 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:58.978807 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:58.978807 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:58.978807 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:58.978807 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:58.978807 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:58.978807 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:58.979858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 73426422-a7fc-44cd-9fc5-cd2d8fa95c38 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:58.979858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:58.979858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:35:58.979858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:35:58.979858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:35:58.979858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:35:58.979858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:35:58.979858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:58.979858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:58.979858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:58.979858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:58.980150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:58.980150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:58.980150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:58.980150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:58.980150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:58.980150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:58.980150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:58.980150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:58.980150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:58.980150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:58.980150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:58.980150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:58.980150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:58.980150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:58.980495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:58.980495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:58.980495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:58.980495 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 4dbcf6ec-b714-44e9-bf1a-8ee4997dc090 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:35:59.013597 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-4dbcf6ec-b714-44e9-bf1a-8ee4997dc090, resource={'id': '4dbcf6ec-b714-44e9-bf1a-8ee4997dc090', 'name': 'tempest-TestEncryptedCinderVolumes-1962176711-network', 'tenant_id': '3152b8f1077d43ebbbf6414c78c2fc79', 'project_id': '3152b8f1077d43ebbbf6414c78c2fc79', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['c331d827-f056-48c7-8287-835ccc1d16c5'], 'standard_attr_id': 1216, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:32:49Z', 'updated_at': '2026-06-08T14:32:49Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 61475}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.013856 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-4dbcf6ec-b714-44e9-bf1a-8ee4997dc090, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestEncryptedCinderVolumes-1962176711-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.014132 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-b5c236b8-1ec2-4597-86b7-bf68e64aca17, external_ids_update=None, columns={'tag': 61475}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.014246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.014723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:59.014723 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:59.014723 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:59.014723 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:59.014723 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:59.014723 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:59.014723 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.014723 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:59.015670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 4dbcf6ec-b714-44e9-bf1a-8ee4997dc090 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.015670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:59.015670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:35:59.015670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:35:59.015670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:35:59.015670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:35:59.015670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:35:59.015670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:59.015670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:59.015670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:59.015670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:59.015990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:59.015990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:59.015990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:59.015990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:59.015990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:59.015990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:59.015990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:59.015990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:59.015990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:59.015990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:59.015990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:59.015990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:59.015990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:59.015990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:59.016440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:59.016440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.016440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:59.016440 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource ec71dd35-9b70-43c3-ba16-6059618c8268 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:35:59.037621 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-ec71dd35-9b70-43c3-ba16-6059618c8268, resource={'id': 'ec71dd35-9b70-43c3-ba16-6059618c8268', 'name': 'tempest-TestEncryptedCinderVolumes-1368072350-network', 'tenant_id': '6fb031a7174d48ca830df3c8feef42ce', 'project_id': '6fb031a7174d48ca830df3c8feef42ce', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['7755c696-ae0d-4e04-9211-5b989bd00961'], 'standard_attr_id': 1230, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:32:55Z', 'updated_at': '2026-06-08T14:32:55Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 24060}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.037899 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-ec71dd35-9b70-43c3-ba16-6059618c8268, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestEncryptedCinderVolumes-1368072350-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.038132 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-f925b94e-fa3e-467b-99d0-d98f1fe7b5c0, external_ids_update=None, columns={'tag': 24060}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.038224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.038681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:59.038681 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:59.038681 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:59.038681 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:59.038681 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:59.038681 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:59.038681 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.038681 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:59.039515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource ec71dd35-9b70-43c3-ba16-6059618c8268 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.039515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:59.039515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:35:59.039515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:35:59.039515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:35:59.039515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:35:59.039515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:35:59.039515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:59.039515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:59.039515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:59.039515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:59.039838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:59.039838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:59.039838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:59.039838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:59.039838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:59.039838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:59.039838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:59.039838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:59.039838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:59.039838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:59.039838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:59.039838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:59.039838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:59.039838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:59.040172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:59.040172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.040172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:59.040172 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource a25665e1-227a-4949-8ee2-7bc2da5f3337 (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:35:59.103134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, resource={'id': 'a25665e1-227a-4949-8ee2-7bc2da5f3337', 'name': 'tempest-TestEncryptedCinderVolumes-1962176711-router', 'project_id': '3152b8f1077d43ebbbf6414c78c2fc79', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.230'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::1a8'}], 'enable_snat': True}, 'gw_port_id': '920851db-ff58-42c4-8d2b-3fcfab52c68a', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.230', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::1a8', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:32:50Z', 'updated_at': '2026-06-08T14:32:52Z', 'revision_number': 3, 'tenant_id': '3152b8f1077d43ebbbf6414c78c2fc79'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.103424 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, columns={'external_ids': {'neutron:router_name': 'tempest-TestEncryptedCinderVolumes-1962176711-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.103625 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "920851db-ff58-42c4-8d2b-3fcfab52c68a"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.103773 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "920851db-ff58-42c4-8d2b-3fcfab52c68a"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.103898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.104451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:59.104451 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:59.104451 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:59.104451 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:59.104451 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:59.104451 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:59.104451 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.104451 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:59.104658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router a25665e1-227a-4949-8ee2-7bc2da5f3337. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.105679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource a25665e1-227a-4949-8ee2-7bc2da5f3337 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.105679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:59.105679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:35:59.105679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:35:59.105679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:35:59.105679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:35:59.105679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:35:59.105679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:35:59.105679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:35:59.105679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:35:59.105679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:35:59.106010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:35:59.106010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:35:59.106010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:59.106010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:59.106010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:59.106010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:59.106010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:59.106010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:59.106010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:59.106010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:59.106010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:59.106010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:59.106010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:59.106010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:59.106360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:59.106360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:59.106360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:59.106360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:59.106360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:59.106360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:59.106360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:59.106360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.106360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:59.106360 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 2cbd0c22-7c21-4b6d-94ed-2faf56386bff (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:35:59.187099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, resource={'id': '2cbd0c22-7c21-4b6d-94ed-2faf56386bff', 'name': 'tempest-TestEncryptedCinderVolumes-1368072350-router', 'project_id': '6fb031a7174d48ca830df3c8feef42ce', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.80'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::381'}], 'enable_snat': True}, 'gw_port_id': '0c296687-b996-4844-bd7e-67b87e0698ed', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.80', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::381', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:32:56Z', 'updated_at': '2026-06-08T14:32:58Z', 'revision_number': 3, 'tenant_id': '6fb031a7174d48ca830df3c8feef42ce'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.187357 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, columns={'external_ids': {'neutron:router_name': 'tempest-TestEncryptedCinderVolumes-1368072350-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.187596 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "0c296687-b996-4844-bd7e-67b87e0698ed"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.187766 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "0c296687-b996-4844-bd7e-67b87e0698ed"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.187908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.188532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:59.188532 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:59.188532 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:59.188532 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:59.188532 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:59.188532 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:59.188532 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.188532 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:59.188736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router 2cbd0c22-7c21-4b6d-94ed-2faf56386bff. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.190396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 2cbd0c22-7c21-4b6d-94ed-2faf56386bff (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.190396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:59.190396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:35:59.190396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:35:59.190396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:35:59.190396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:35:59.190396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:35:59.190396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:35:59.190396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:35:59.190396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:35:59.190396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:35:59.190711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:35:59.190711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:35:59.190711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:59.190711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:59.190711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:59.190711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:59.190711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:59.190711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:59.190711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:59.190711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:59.190711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:59.190711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:59.190711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:59.190711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:59.191071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:59.191071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:59.191071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:59.191071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:59.191071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:59.191071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:59.191071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:59.191071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.191071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:59.191071 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 920851db-ff58-42c4-8d2b-3fcfab52c68a (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:35:59.280305 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-920851db-ff58-42c4-8d2b-3fcfab52c68a, resource={'id': '920851db-ff58-42c4-8d2b-3fcfab52c68a', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:a2:3d:56', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'a25665e1-227a-4949-8ee2-7bc2da5f3337', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1221, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.230'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::1a8'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:32:50Z', 'updated_at': '2026-06-08T14:32:53Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.280597 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-920851db-ff58-42c4-8d2b-3fcfab52c68a, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::1a8/64', '172.24.5.230/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.280919 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=920851db-ff58-42c4-8d2b-3fcfab52c68a, lrouter_port=lrp-920851db-ff58-42c4-8d2b-3fcfab52c68a, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.281135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.281701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:59.281701 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:59.281701 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:59.281701 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:59.281701 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:59.281701 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:59.281701 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.281701 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:59.282664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 920851db-ff58-42c4-8d2b-3fcfab52c68a (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.282664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:59.282664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:35:59.282664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:35:59.282664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:35:59.282664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:35:59.282664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:35:59.282664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:59.282664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:59.282664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:59.282664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:59.282926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:59.282926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:59.282926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:59.282926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:59.282926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:59.282926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:59.282926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:59.282926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:59.282926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:59.282926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:59.282926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:59.282926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:59.282926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:59.282926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:59.283175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:59.283175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.283175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:59.283175 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 0c296687-b996-4844-bd7e-67b87e0698ed (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:35:59.369556 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-0c296687-b996-4844-bd7e-67b87e0698ed, resource={'id': '0c296687-b996-4844-bd7e-67b87e0698ed', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:3a:85:43', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '2cbd0c22-7c21-4b6d-94ed-2faf56386bff', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1235, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.80'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::381'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:32:56Z', 'updated_at': '2026-06-08T14:32:58Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.369905 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-0c296687-b996-4844-bd7e-67b87e0698ed, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::381/64', '172.24.5.80/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.370198 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=0c296687-b996-4844-bd7e-67b87e0698ed, lrouter_port=lrp-0c296687-b996-4844-bd7e-67b87e0698ed, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:35:59.370438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.370929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:35:59.370929 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:59.370929 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:35:59.370929 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:35:59.370929 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:59.370929 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:35:59.370929 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.370929 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:35:59.371893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 0c296687-b996-4844-bd7e-67b87e0698ed (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.371893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:35:59.371893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:35:59.371893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:35:59.371893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:35:59.371893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:35:59.371893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:35:59.371893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:35:59.371893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:59.371893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:59.371893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:35:59.372102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:35:59.372102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:35:59.372102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:35:59.372102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:35:59.372102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:35:59.372102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:35:59.372102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:35:59.372102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:35:59.372102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:35:59.372102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:35:59.372102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:35:59.372102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:35:59.372102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:35:59.372102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:35:59.372383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:35:59.372383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:35:59.372383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:35:59.372489 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 23d8feb6-6004-41e4-a5fe-404b0919d121 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:35:59.372703 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(23d8feb6-6004-41e4-a5fe-404b0919d121) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:35:59.378695 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 39a2aca3-d443-4286-981d-4ec80246b335 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:35:59.379522 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(39a2aca3-d443-4286-981d-4ec80246b335) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:35:59.387331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 3c203229-1bb6-4b5b-9341-f04e978ff668 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:35:59.387605 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(3c203229-1bb6-4b5b-9341-f04e978ff668) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:35:59.392726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 52338674-2ab6-4c4d-b00a-6f7a6eb5ffe3 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:35:59.393028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(52338674-2ab6-4c4d-b00a-6f7a6eb5ffe3) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:35:59.397285 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 63982487-c78c-4d99-a277-6985e6e8e6fb (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:35:59.397562 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(63982487-c78c-4d99-a277-6985e6e8e6fb) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:35:59.401929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 7f3d2499-fd49-4ffb-9b50-92cf8a4739d5 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:35:59.402170 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(7f3d2499-fd49-4ffb-9b50-92cf8a4739d5) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:35:59.407628 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource a0249088-ce50-4289-927a-b0142e1aca5d (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:35:59.407861 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(a0249088-ce50-4289-927a-b0142e1aca5d) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:35:59.416371 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource b09910e6-6878-40cb-a995-7f3c9a043709 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:35:59.416611 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(b09910e6-6878-40cb-a995-7f3c9a043709) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:35:59.422374 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource c7572632-811e-4f9a-90a8-1abac64a4bdc (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:35:59.422554 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(c7572632-811e-4f9a-90a8-1abac64a4bdc) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:35:59.429124 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource dab47abd-ef9d-4701-9831-0872332585b4 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:35:59.429484 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(dab47abd-ef9d-4701-9831-0872332585b4) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:35:59.439312 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource e15c0ec6-020b-4475-81bf-582cec4587be (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:35:59.439529 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(e15c0ec6-020b-4475-81bf-582cec4587be) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:35:59.448954 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource e65a9435-2cf0-4ba7-a341-847105facc00 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:35:59.449165 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(e65a9435-2cf0-4ba7-a341-847105facc00) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:35:59.459424 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource e9ce57d5-c4a1-4ec8-9c84-2fc72208fb6a (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:35:59.459424 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(e9ce57d5-c4a1-4ec8-9c84-2fc72208fb6a) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:35:59.462886 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource f81cc546-e1a3-499a-b700-405a8e7bb66b (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:35:59.463183 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(f81cc546-e1a3-499a-b700-405a8e7bb66b) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:35:59.470983 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource f8c22fde-88ff-45d3-b601-a8438ea13038 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:35:59.471214 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(f8c22fde-88ff-45d3-b601-a8438ea13038) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:35:59.480580 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource fbc52714-db33-48f2-a085-ba4af01e2a39 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:35:59.480856 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(fbc52714-db33-48f2-a085-ba4af01e2a39) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:35:59.486813 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_for_inconsistencies finished in 0.627 seconds Jun 08 14:36:00.824036 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:00.824537 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:00.824537 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:00.825070 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:00.825234 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:00.825382 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:00.825511 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:00.825606 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:00.825804 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:00.825804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.826273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:00.826273 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:00.826273 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:00.826273 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:00.826273 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:00.826273 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:00.826273 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.826273 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:00.827281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.827281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:00.827281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:00.827281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:00.827281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.827281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:00.827281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:00.827281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.827281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:00.827281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:00.827281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:00.827281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.827680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:00.827680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:00.827680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:00.827680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:00.827680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:00.827680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:00.827680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:00.827680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:00.827680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:00.827680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:00.827680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:00.827680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:00.827680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:00.827997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.827997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:00.828268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.828268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:00.828268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:00.828268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:00.828268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:00.828268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:00.828268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.828268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:00.828268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:00.828268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.828268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:00.828268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:00.828268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.828268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:00.828268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:00.828268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.828684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:00.828684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:00.828684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.828684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:00.828684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:00.828684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:00.828684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.828684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:00.828684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:00.828684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:00.828684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:00.828684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:00.828684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:00.828684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:00.828684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:00.828684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:00.828684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:00.828684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:00.828684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:00.829177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:00.829177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.829177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:00.836367 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:00.837034 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:00.837122 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:00.837218 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:36:00.837573 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:00.837799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.838545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:00.838545 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:00.838545 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:00.838545 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:00.838545 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:00.838545 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:00.838545 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.838545 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:00.839171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.839171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:00.839171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:00.839171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:00.839171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.839171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:00.839171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:00.839171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:00.839171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.839171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:00.839171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:00.839171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:00.839171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:00.839171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:00.839171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:00.839596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:00.839596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:00.839596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:00.839596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:00.839596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:00.839596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:00.839596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:00.839596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.839596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:00.841084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.841084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:00.841084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:00.841084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:00.841084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.841084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:00.841084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:00.841084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:36:00.841084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:00.841084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:00.841084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.841084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:00.841566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:00.841566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:00.841566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.841566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:00.841566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:00.841566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:00.841566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:00.841566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:00.841566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:00.841566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:00.841566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:00.841566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:00.841566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:00.841566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:00.842097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:00.842097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:00.842097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.842097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:00.842406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.842406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:00.842406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:00.842406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:00.842406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:00.842406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:00.842406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.842406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:00.842406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:00.842406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.842406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:00.842406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:00.842406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.842406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:00.842406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:00.842406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:00.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:00.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:36:00.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:00.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:00.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:00.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:00.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:00.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:00.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:00.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:00.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:00.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:00.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:00.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:00.842956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:00.843475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:00.843475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:00.843475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:00.843475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:00.843475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:00.843475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.843475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:00.844640 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:00.844907 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:00.844960 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:00.845175 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:00.845438 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:00.845579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.845996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:00.845996 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:00.845996 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:00.845996 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:00.845996 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:00.845996 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:00.845996 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.845996 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:00.846538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.846538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:00.846538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:00.846538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:00.846538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.846538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:00.846538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:00.846538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:00.846538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.846538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:00.846538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:00.846538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:00.846538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:00.846538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:00.846538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:00.846986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:00.846986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:00.846986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:00.846986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:00.846986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:00.846986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:00.846986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:00.846986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.846986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:00.847402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.847402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:00.847402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:00.847402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:00.847402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.847402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:00.847402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:00.847402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:00.847402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:00.847402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.847402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:00.847731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:00.847731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:00.847731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.847731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:00.847731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:00.847731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:00.847731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:00.847731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:00.847731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:00.847731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:00.847731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:00.847731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:00.847731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:00.847731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:00.848156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:00.848156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:00.848156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.848156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:00.848156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.848156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:00.848156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:00.848156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:00.848156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:00.848156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:00.848156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.848156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:00.848156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:00.848156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.848570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:00.848570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:00.848570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.848570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:00.848570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:00.848570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.848570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:00.848570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:00.848570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:00.848570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:00.848570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.848570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:00.848570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:00.848570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:00.848570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.848570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:00.848570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:00.848570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:00.848570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:00.849124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:00.849124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:00.849124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:00.849124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:00.849124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:00.849124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:00.849124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:00.849124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:00.849124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:00.849124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.849124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:00.869985 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:00.870261 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:00.870390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:00.870571 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:00.871694 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:00.871887 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:00.872047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.872460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:00.872460 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:00.872460 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:00.872460 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:00.872460 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:00.872460 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:00.872460 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.872460 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:00.873305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.873305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:00.873305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:00.873305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:00.873305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.873305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:00.873305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:00.873305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.873305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:00.873305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:00.873305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:00.873305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.873514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:00.873514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:00.873514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:00.873514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:00.873514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:00.873514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:00.873514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:00.873514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:00.873514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:00.873514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:00.873514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:00.873514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:00.873514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:00.873730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.873730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:00.874125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.874125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:00.874125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:00.874125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:00.874125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:00.874125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:00.874125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.874125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:00.874125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:00.874125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.874125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:00.874125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:00.874125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.874125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:00.874125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:00.874125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:00.874400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:00.874400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:00.874400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.874400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:00.874400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:00.874400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:00.874400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:00.874400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:00.874400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:00.874400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:00.874400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:00.874400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:00.874400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:00.874400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:00.874400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:00.874400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:00.874400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:00.874400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:00.874400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:00.874703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:00.874703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:00.874703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:03.743604 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:03.744238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:03.744238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:03.744646 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:03.789959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:03.790199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:03.790731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:03.790731 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:03.790731 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:03.790731 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:03.790731 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:03.790731 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:03.790731 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:03.790731 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:03.791665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:03.791665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:03.791665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:03.791665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:03.791665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:03.791665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:03.791665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:03.791665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:03.791665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:03.791665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:03.791665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:03.791665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:03.792023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:03.792023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:03.792023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:03.792023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:03.792023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:03.792023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:03.792023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:03.792023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:03.792023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:03.792023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:03.792023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:03.792023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:03.792023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:03.792453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:03.792453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:03.794873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:03.794873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:03.794873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:03.794873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:03.794873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:03.794873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:03.794873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:03.794873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:03.794873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:03.794873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:03.794873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:03.794873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:03.794873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:03.794873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:03.794873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:03.794873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:03.795393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:03.795393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:03.795393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:03.795393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:03.795393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:03.795393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:03.795393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:03.795393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:03.795393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:03.795393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:03.795393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:03.795393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:03.795393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:03.795393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:03.795393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:03.795393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:03.795393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:03.795393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:03.795393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:03.795925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:03.795925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:03.795925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:05.824520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:05.824994 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:05.824994 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:05.825539 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:05.825679 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:05.825857 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:05.825996 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:05.826024 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:05.826130 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:05.826234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.826656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:05.826656 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:05.826656 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:05.826656 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:05.826656 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:05.826656 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:05.826656 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.826656 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:05.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:05.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:05.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:05.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:05.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:05.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:05.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:05.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:05.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.827561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:05.827561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:05.827561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:05.827561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:05.827561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:05.827561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:05.827561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:05.827561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:05.827561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:05.827561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:05.827561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:05.827561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:05.827561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:05.827772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.827772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:05.828061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.828061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:05.828061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:05.828061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:05.828061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:05.828061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:05.828061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.828061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:05.828061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:05.828061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.828061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:05.828061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:05.828061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.828061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:05.828061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:05.828061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.828648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:05.828648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:05.828648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.828648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:05.828648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:05.828648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:05.828648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.828648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:05.828648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:05.828648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:05.828648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:05.828648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:05.828648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:05.828648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:05.828648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:05.828648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:05.828648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:05.828648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:05.828648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:05.829336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:05.829336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.829336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:05.837105 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:05.837366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:05.837460 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:05.837544 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:36:05.837859 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:05.838014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.838427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:05.838427 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:05.838427 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:05.838427 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:05.838427 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:05.838427 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:05.838427 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.838427 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:05.838981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.838981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:05.838981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:05.838981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:05.838981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.838981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:05.838981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:05.838981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:05.838981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.838981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:05.838981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:05.838981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:05.838981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:05.838981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:05.838981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:05.839278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:05.839278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:05.839278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:05.839278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:05.839278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:05.839278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:05.839278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:05.839278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.839278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:05.839654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.839654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:05.839654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:05.839654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:05.839654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.839654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:05.839654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:05.839654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:36:05.839654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:05.839654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:05.839654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.839654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:05.840402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:05.840402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:05.840402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.840402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:05.840402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:05.840402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:05.840402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:05.840402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:05.840402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:05.840402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:05.840402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:05.840402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:05.840402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:05.840402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:05.840757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:05.840757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:05.840757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.840757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:05.840757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.840757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:05.840757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:05.840757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:05.840757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:05.840757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:05.840757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.840757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:05.840757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:05.840757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.840987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:05.840987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:05.840987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.840987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:05.840987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:05.840987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.840987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:05.840987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:05.840987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:36:05.840987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:05.840987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:05.840987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.840987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:05.840987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:05.840987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:05.840987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.840987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:05.840987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:05.840987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:05.841333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:05.841333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:05.841333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:05.841333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:05.841333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:05.841333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:05.841333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:05.841333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:05.841333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:05.841333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:05.841333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.841333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:05.844928 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:05.845132 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:05.845219 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:05.845339 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:05.845599 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:05.845810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:05.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:05.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:05.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:05.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:05.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:05.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:05.846660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.846660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:05.846660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:05.846660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:05.846660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.846660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:05.846660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:05.846660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:05.846660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.846660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:05.846660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:05.846660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:05.846660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:05.846660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:05.846660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:05.846922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:05.846922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:05.846922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:05.846922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:05.846922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:05.846922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:05.846922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:05.846922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.846922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:05.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:05.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:05.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:05.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:05.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:05.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:05.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:05.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:05.847782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:05.847782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:05.847782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.847782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:05.847782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:05.847782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:05.847782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:05.847782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:05.847782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:05.847782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:05.847782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:05.847782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:05.847782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:05.847782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:05.848188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:05.848188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:05.848188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.848188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:05.848188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.848188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:05.848188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:05.848188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:05.848188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:05.848188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:05.848188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.848188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:05.848188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:05.848188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.848475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:05.848475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:05.848475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.848475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:05.848475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:05.848475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.848475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:05.848475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:05.848475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:05.848475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:05.848475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.848475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:05.848475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:05.848475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:05.848475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.848475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:05.848475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:05.848475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:05.848475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:05.848781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:05.848781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:05.848781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:05.848781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:05.848781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:05.848781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:05.848781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:05.848781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:05.848781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:05.848781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.848781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:05.870387 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:05.870680 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:05.870743 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:05.870886 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:05.871169 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:05.871399 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:05.871569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.872024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:05.872024 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:05.872024 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:05.872024 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:05.872024 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:05.872024 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:05.872024 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.872024 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:05.872631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.872631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:05.872631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:05.872631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:05.872631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.872631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:05.872631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:05.872631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.872631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:05.872631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:05.872631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:05.872631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.872995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:05.872995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:05.872995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:05.872995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:05.872995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:05.872995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:05.872995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:05.872995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:05.872995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:05.872995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:05.872995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:05.872995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:05.872995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:05.873332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.873332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:05.873332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.873332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:05.873332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:05.873332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:05.873332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:05.873332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:05.873332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.873332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:05.873332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:05.873332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.873332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:05.873332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:05.873332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.873748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:05.873748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:05.873748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:05.873748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:05.873748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:05.873748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.873748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:05.873748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:05.873748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:05.873748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:05.873748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:05.873748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:05.873748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:05.873748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:05.873748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:05.873748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:05.873748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:05.873748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:05.873748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:05.874177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:05.874177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:05.874177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:05.874177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:05.874177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:05.874177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:08.744120 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:08.744583 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:08.744583 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:08.745187 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:08.774390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:08.774601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:08.775096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:08.775096 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:08.775096 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:08.775096 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:08.775096 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:08.775096 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:08.775096 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:08.775096 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:08.775805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:08.775805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:08.775805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:08.775805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:08.775805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:08.775805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:08.775805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:08.775805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:08.775805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:08.775805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:08.775805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:08.775805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:08.776034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:08.776034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:08.776034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:08.776034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:08.776034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:08.776034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:08.776034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:08.776034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:08.776034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:08.776034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:08.776034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:08.776034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:08.776034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:08.776236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:08.776236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:08.776546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:08.776546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:08.776546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:08.776546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:08.776546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:08.776546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:08.776546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:08.776546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:08.776546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:08.776546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:08.776546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:08.776546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:08.776546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:08.776546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:08.776546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:08.776546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:08.776828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:08.776828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:08.776828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:08.776828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:08.776828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:08.776828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:08.776828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:08.776828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:08.776828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:08.776828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:08.776828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:08.776828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:08.776828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:08.776828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:08.776828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:08.776828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:08.776828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:08.776828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:08.776828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:08.777107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:08.777107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:08.777107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:10.824867 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:10.826644 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:10.826768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:10.827654 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:10.827830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:10.827948 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:10.828049 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:10.828146 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:10.828248 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:10.828366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.828902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:10.828902 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:10.828902 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:10.828902 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:10.828902 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:10.828902 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:10.828902 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.828902 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:10.830063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.830063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:10.830063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:10.830063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:10.830063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.830063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:10.830063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:10.830063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.830063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:10.830063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:10.830063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:10.830063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.830282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:10.830282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:10.830282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:10.830282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:10.830282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:10.830282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:10.830282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:10.830282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:10.830282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:10.830282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:10.830282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:10.830282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:10.830282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:10.830493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.830493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:10.832201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.832201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:10.832201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:10.832201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:10.832201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:10.832201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:10.832201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.832201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:10.832201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:10.832201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.832201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:10.832201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:10.832201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.832201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:10.832201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:10.832201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.832476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:10.832476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:10.832476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.832476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:10.832476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:10.832476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:10.832476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.832476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:10.832476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:10.832476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:10.832476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:10.832476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:10.832476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:10.832476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:10.832476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:10.832476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:10.832476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:10.832476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:10.832476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:10.832790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:10.832790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.832790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:10.837484 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:10.837942 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:10.838163 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:10.838702 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:36:10.839078 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:10.839233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.840115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:10.840115 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:10.840115 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:10.840115 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:10.840115 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:10.840115 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:10.840115 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.840115 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:10.840978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.840978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:10.840978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:10.840978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:10.840978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.840978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:10.840978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:10.840978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:10.840978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.840978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:10.840978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:10.840978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:10.840978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:10.840978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:10.840978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:10.841232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:10.841232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:10.841232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:10.841232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:10.841232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:10.841232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:10.841232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:10.841232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.841232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:10.841894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.841894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:10.841894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:10.841894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:10.841894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.841894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:10.841894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:10.841894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:36:10.841894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:10.841894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:10.841894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.841894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:10.842213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:10.842213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:10.842213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.842213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:10.842213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:10.842213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:10.842213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:10.842213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:10.842213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:10.842213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:10.842213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:10.842213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:10.842213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:10.842213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:10.842477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:10.842477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:10.842477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.842477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:10.845183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.845183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:10.845183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:10.845183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:10.845183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:10.845183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:10.845183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.845183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:10.845183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:10.845183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.845183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:10.845183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:10.845183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.845183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:10.845183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:10.845183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.845484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:10.845484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:10.845484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:36:10.845484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:10.845484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:10.845484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.845484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:10.845484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:10.845484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:10.845484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.845484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:10.845484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:10.845484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:10.845484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:10.845484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:10.845484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:10.845484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:10.845484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:10.845814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:10.845814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:10.845814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:10.845814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:10.845814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:10.845814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.845814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:10.845814 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:10.845975 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:10.846380 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:10.846607 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:10.847065 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:10.847303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.847749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:10.847749 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:10.847749 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:10.847749 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:10.847749 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:10.847749 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:10.847749 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.847749 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:10.848548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.848548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:10.848548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:10.848548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:10.848548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.848548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:10.848548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:10.848548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:10.848548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.848548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:10.848548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:10.848548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:10.848548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:10.848548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:10.848548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:10.848830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:10.848830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:10.848830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:10.848830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:10.848830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:10.848830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:10.848830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:10.848830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.848830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:10.849441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.849441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:10.849441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:10.849441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:10.849441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.849441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:10.849441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:10.849441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:10.849441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:10.849441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.849441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:10.849637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:10.849637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:10.849637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.849637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:10.849637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:10.849637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:10.849637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:10.849637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:10.849637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:10.849637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:10.849637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:10.849637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:10.849637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:10.849637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:10.849955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:10.849955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:10.849955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.849955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:10.852735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.852735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:10.852735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:10.852735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:10.852735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:10.852735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:10.852735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.852735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:10.852735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:10.852735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.852735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:10.852735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:10.852735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.852735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:10.852735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:10.852735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.852735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:10.853014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:10.853014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:10.853014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:10.853014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.853014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:10.853014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:10.853014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:10.853014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.853014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:10.853014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:10.853014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:10.853014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:10.853014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:10.853014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:10.853014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:10.853014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:10.853014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:10.853014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:10.853014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:10.853386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:10.853386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:10.853386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.853386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:10.873461 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:10.873461 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:10.873461 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:10.873461 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:10.873461 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:10.873461 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:10.873708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.873708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:10.873708 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:10.873708 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:10.873708 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:10.873708 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:10.873708 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:10.873708 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.873708 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:10.873708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.873708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:10.873708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:10.873708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:10.873708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.873708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:10.874006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:10.874006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.874006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:10.874006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:10.874006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:10.874006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.874006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:10.874006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:10.874006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:10.874006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:10.874006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:10.874006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:10.874006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:10.874006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:10.874240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:10.874240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:10.874240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:10.874240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:10.874240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:10.874240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.874240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:10.874240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.874240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:10.874240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:10.874240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:10.874240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:10.874240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:10.874482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:10.874805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:10.874805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:10.874805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:10.874805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:10.874805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:10.874805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:10.874805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:10.874805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:10.874805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:10.874805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:10.874805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:10.874805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:13.745218 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:13.747043 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:13.747108 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:13.747986 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:13.813657 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:13.814069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:13.814709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:13.814709 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:13.814709 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:13.814709 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:13.814709 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:13.814709 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:13.814709 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:13.814709 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:13.815433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.068 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:13.815433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:13.815433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:13.815433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:13.815433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:13.815433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:13.815433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:13.815433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:13.815433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:13.815433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:13.815433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:13.815433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:13.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:13.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:13.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:13.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:13.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:13.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:13.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:13.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:13.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:13.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:13.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:13.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:13.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:13.816065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:13.816065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:13.816199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:13.816199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:13.816199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:13.816199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:13.816199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:13.816199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:13.816199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:13.816199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:13.816199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:13.816199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:13.816199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:13.816199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:13.816199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:13.816199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:13.816199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:13.816199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:13.816600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:13.816600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:13.816600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:13.816600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:13.816600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:13.816600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:13.816600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:13.816600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:13.816600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:13.816600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:13.816600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:13.816600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:13.816600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:13.816600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:13.816600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:13.816600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:13.816600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:13.816600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:13.816600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:13.816965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:13.816965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:13.816965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:15.826681 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:15.827134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:15.827134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:15.827790 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:15.827913 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:15.828024 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:15.828119 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:15.828208 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:15.828329 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:15.828419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.828848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:15.828848 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:15.828848 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:15.828848 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:15.828848 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:15.828848 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:15.828848 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.828848 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:15.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:15.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:15.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:15.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:15.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:15.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:15.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:15.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:15.829441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.829813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:15.829813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:15.829813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:15.829813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:15.829813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:15.829813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:15.829813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:15.829813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:15.829813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:15.829813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:15.829813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:15.829813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:15.829813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:15.830118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.830118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:15.830118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.830118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:15.830118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:15.830118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:15.830118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:15.830118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:15.830118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.830118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:15.830118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:15.830118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.830118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:15.830118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:15.830118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.830575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:15.830575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:15.830575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.830575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:15.830575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:15.830575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.830575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:15.830575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:15.830575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:15.830575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.830575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:15.830575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:15.830575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:15.830575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:15.830575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:15.830575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:15.830575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:15.830575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:15.830575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:15.831222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:15.831222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:15.831222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:15.831222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:15.831222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.831222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:15.837927 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:15.838113 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:15.838192 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:15.838319 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:36:15.838548 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:15.838714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.839057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:15.839057 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:15.839057 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:15.839057 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:15.839057 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:15.839057 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:15.839057 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.839057 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:15.839528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.839528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:15.839528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:15.839528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:15.839528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.839528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:15.839528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:15.839528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:15.839528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.839528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:15.839528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:15.839528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:15.839528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:15.839528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:15.839528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:15.840017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:15.840017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:15.840017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:15.840017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:15.840017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:15.840017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:15.840017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:15.840017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.840017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:15.840324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.840324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:15.840324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:15.840324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:15.840324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.840324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:15.840324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:15.840324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:36:15.840324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:15.840324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:15.840324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.840324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:15.840711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:15.840711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:15.840711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.840711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:15.840711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:15.840711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:15.840711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:15.840711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:15.840711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:15.840711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:15.840711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:15.840711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:15.840711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:15.840711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:15.841178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:15.841178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:15.841178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.841178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:15.841178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.841178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:15.841178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:15.841178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:15.841178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:15.841178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:15.841178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.841178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:15.841178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:15.841178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.841659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:15.841659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:15.841659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.841659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:15.841659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:15.841659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.841659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:15.841659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:15.841659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:36:15.841659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:15.841659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:15.841659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.841659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:15.841659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:15.841659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:15.841659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.841659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:15.841659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:15.841659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:15.842218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:15.842218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:15.842218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:15.842218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:15.842218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:15.842218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:15.842218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:15.842218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:15.842218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:15.842218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:15.842218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.842218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:15.846050 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:15.846306 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:15.846398 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:15.846489 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:15.846724 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:15.846889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.847194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:15.847194 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:15.847194 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:15.847194 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:15.847194 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:15.847194 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:15.847194 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.847194 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:15.847685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.847685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:15.847685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:15.847685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:15.847685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.847685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:15.847685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:15.847685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:15.847685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.847685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:15.847685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:15.847685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:15.847685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:15.847685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:15.847685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:15.848119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:15.848119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:15.848119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:15.848119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:15.848119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:15.848119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:15.848119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:15.848119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.848119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:15.848409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.848409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:15.848409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:15.848409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:15.848409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.848409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:15.848409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:15.848409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:15.848409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:15.848409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.848409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:15.848625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:15.848625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:15.848625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.848625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:15.848625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:15.848625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:15.848625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:15.848625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:15.848625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:15.848625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:15.848625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:15.848625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:15.848625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:15.848625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:15.848838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:15.848838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:15.848838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.848838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:15.848965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.848965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:15.848965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:15.848965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:15.848965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:15.848965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:15.848965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.848965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:15.848965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:15.848965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.848965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:15.848965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:15.848965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.848965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:15.848965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:15.848965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.848965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:15.849246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:15.849246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:15.849246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:15.849246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.849246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:15.849246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:15.849246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:15.849246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.849246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:15.849246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:15.849246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:15.849246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:15.849246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:15.849246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:15.849246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:15.849246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:15.849246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:15.849246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:15.849246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:15.849545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:15.849545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:15.849545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.849545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:15.871278 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:15.871415 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:15.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:15.871690 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:15.871937 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:15.872116 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:15.872295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.872699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:15.872699 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:15.872699 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:15.872699 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:15.872699 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:15.872699 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:15.872699 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.872699 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:15.873261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.873261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:15.873261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:15.873261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:15.873261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.873261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:15.873261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:15.873261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.873261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:15.873261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:15.873261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:15.873261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.873617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:15.873617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:15.873617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:15.873617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:15.873617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:15.873617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:15.873617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:15.873617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:15.873617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:15.873617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:15.873617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:15.873617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:15.873617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:15.874033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.874033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:15.874033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.874033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:15.874033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:15.874033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:15.874033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:15.874033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:15.874033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.874033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:15.874033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:15.874033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.874033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:15.874033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:15.874033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.874581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:15.874581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:15.874581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:15.874581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:15.874581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:15.874581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.874581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:15.874581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:15.874581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:15.874581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:15.874581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:15.874581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:15.874581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:15.874581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:15.874581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:15.874581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:15.874581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:15.874581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:15.874581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:15.875165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:15.875165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:15.875165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:15.875165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:15.875165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:15.875165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:18.745820 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:18.746317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:18.746317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:18.746751 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:18.781426 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:18.781603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:18.782054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:18.782054 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:18.782054 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:18.782054 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:18.782054 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:18.782054 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:18.782054 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:18.782054 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:18.782892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:18.782892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:18.782892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:18.782892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:18.782892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:18.782892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:18.782892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:18.782892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:18.782892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:18.782892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:18.782892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:18.782892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:18.783130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:18.783130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:18.783130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:18.783130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:18.783130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:18.783130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:18.783130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:18.783130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:18.783130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:18.783130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:18.783130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:18.783130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:18.783130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:18.783373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:18.783373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:18.783836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:18.783836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:18.783836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:18.783836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:18.783836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:18.783836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:18.783836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:18.783836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:18.783836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:18.783836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:18.783836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:18.783836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:18.783836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:18.783836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:18.783836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:18.783836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:18.784118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:18.784118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:18.784118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:18.784118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:18.784118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:18.784118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:18.784118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:18.784118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:18.784118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:18.784118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:18.784118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:18.784118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:18.784118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:18.784118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:18.784118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:18.784118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:18.784118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:18.784118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:18.784118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:18.784416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:18.784416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:18.784416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:20.827076 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:20.827645 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:20.827645 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:20.828295 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:20.828509 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:20.828673 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:20.828787 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:20.828891 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:20.828994 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:20.829106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.829602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:20.829602 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:20.829602 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:20.829602 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:20.829602 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:20.829602 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:20.829602 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.829602 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:20.830312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.830312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:20.830312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:20.830312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:20.830312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.830312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:20.830312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:20.830312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.830312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:20.830312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:20.830312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:20.830312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.830549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:20.830549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:20.830549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:20.830549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:20.830549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:20.830549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:20.830549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:20.830549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:20.830549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:20.830549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:20.830549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:20.830549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:20.830549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:20.830767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.830767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:20.831046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.831046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:20.831046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:20.831046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:20.831046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:20.831046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:20.831046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.831046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:20.831046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:20.831046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.831046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:20.831046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:20.831046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.831046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:20.831046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:20.831046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.831342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:20.831342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:20.831342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.831342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:20.831342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:20.831342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:20.831342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.831342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:20.831342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:20.831342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:20.831342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:20.831342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:20.831342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:20.831342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:20.831342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:20.831342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:20.831342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:20.831342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:20.831342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:20.831702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:20.831702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.831702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:20.838218 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:20.838394 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:20.838479 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:20.838585 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:36:20.838883 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:20.839035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.839454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:20.839454 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:20.839454 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:20.839454 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:20.839454 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:20.839454 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:20.839454 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.839454 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:20.839956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.839956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:20.839956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:20.839956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:20.839956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.839956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:20.839956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:20.839956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:20.839956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.839956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:20.839956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:20.839956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:20.839956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:20.839956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:20.839956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:20.840223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:20.840223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:20.840223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:20.840223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:20.840223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:20.840223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:20.840223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:20.840223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.840223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:20.840500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.840500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:20.840500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:20.840500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:20.840500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.840500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:20.840500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:20.840500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:36:20.840500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:20.840500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:20.840500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.840500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:20.840704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:20.840704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:20.840704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.840704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:20.840704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:20.840704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:20.840704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:20.840704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:20.840704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:20.840704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:20.840704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:20.840704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:20.840704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:20.840704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:20.840927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:20.840927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:20.840927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.840927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:20.841171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.841171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:20.841171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:20.841171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:20.841171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:20.841171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:20.841171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.841171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:20.841171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:20.841171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.841171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:20.841171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:20.841171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.841171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:20.841171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:20.841171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.841435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:20.841435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:20.841435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:36:20.841435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:20.841435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:20.841435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.841435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:20.841435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:20.841435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:20.841435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.841435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:20.841435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:20.841435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:20.841435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:20.841435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:20.841435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:20.841435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:20.841435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:20.841756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:20.841756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:20.841756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:20.841756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:20.841756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:20.841756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.841756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:20.846323 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:20.846475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:20.846575 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:20.846684 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:20.846959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:20.847162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.847577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:20.847577 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:20.847577 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:20.847577 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:20.847577 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:20.847577 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:20.847577 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.847577 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:20.848062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.848062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:20.848062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:20.848062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:20.848062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.848062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:20.848062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:20.848062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:20.848062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.848062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:20.848062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:20.848062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:20.848062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:20.848062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:20.848062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:20.848406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:20.848406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:20.848406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:20.848406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:20.848406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:20.848406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:20.848406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:20.848406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.848406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:20.848690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.848690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:20.848690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:20.848690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:20.848690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.848690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:20.848690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:20.848690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:20.848690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:20.848690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.848690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:20.848886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:20.848886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:20.848886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.848886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:20.848886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:20.848886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:20.848886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:20.848886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:20.848886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:20.848886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:20.848886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:20.848886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:20.848886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:20.848886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:20.849134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:20.849134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:20.849134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.849134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:20.849372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.849372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:20.849372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:20.849372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:20.849372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:20.849372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:20.849372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.849372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:20.849372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:20.849372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.849372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:20.849372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:20.849372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.849372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:20.849372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:20.849372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.849372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:20.849836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:20.849836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:20.849836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:20.849836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.849836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:20.849836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:20.849836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:20.849836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.849836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:20.849836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:20.849836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:20.849836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:20.849836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:20.849836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:20.849836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:20.849836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:20.849836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:20.849836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:20.849836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:20.850179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:20.850179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:20.850179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.850179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:20.871534 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:20.871756 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:20.871844 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:20.871975 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:20.872266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:20.872482 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:20.872717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.873134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:20.873134 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:20.873134 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:20.873134 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:20.873134 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:20.873134 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:20.873134 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.873134 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:20.873756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.873756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:20.873756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:20.873756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:20.873756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.873756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:20.873756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:20.873756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.873756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:20.873756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:20.873756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:20.873756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.873974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:20.873974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:20.873974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:20.873974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:20.873974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:20.873974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:20.873974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:20.873974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:20.873974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:20.873974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:20.873974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:20.873974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:20.873974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:20.874218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.874218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:20.874372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.874372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:20.874372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:20.874372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:20.874372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:20.874372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:20.874372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.874372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:20.874372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:20.874372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.874372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:20.874372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:20.874372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.874372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:20.874372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:20.874372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:20.874675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:20.874675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:20.874675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.874675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:20.874675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:20.874675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:20.874675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:20.874675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:20.874675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:20.874675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:20.874675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:20.874675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:20.874675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:20.874675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:20.874675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:20.874675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:20.874675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:20.874675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:20.874675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:20.875181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:20.875181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:20.875181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:23.746555 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:23.747082 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:23.747082 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:23.747384 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:23.782541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:23.782957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:23.783687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:23.783687 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:23.783687 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:23.783687 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:23.783687 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:23.783687 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:23.783687 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:23.783687 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:23.784381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:23.784381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:23.784381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:23.784381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:23.784381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:23.784381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:23.784381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:23.784381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:23.784381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:23.784381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:23.784381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:23.784381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:23.784855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:23.784855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:23.784855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:23.784855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:23.784855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:23.784855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:23.784855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:23.784855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:23.784855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:23.784855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:23.784855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:23.784855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:23.784855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:23.785333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:23.785333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:23.785333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:23.785333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:23.785333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:23.785333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:23.785333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:23.785333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:23.785333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:23.785333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:23.785333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:23.785333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:23.785333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:23.785333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:23.785333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:23.785835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:23.785835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:23.785835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:23.785835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:23.785835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:23.785835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:23.785835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:23.785835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:23.785835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:23.785835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:23.785835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:23.785835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:23.785835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:23.785835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:23.785835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:23.785835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:23.785835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:23.785835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:23.785835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:23.786459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:23.786459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:23.786459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:23.786459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:23.786459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:23.786459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:25.828087 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:25.828087 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:25.828799 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:25.828967 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:25.829061 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:25.829178 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:25.829365 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:25.829501 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:25.829677 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:25.829820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.830239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:25.830239 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:25.830239 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:25.830239 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:25.830239 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:25.830239 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:25.830239 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.830239 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:25.830918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.830918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:25.830918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:25.830918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:25.830918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.830918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:25.830918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:25.830918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.830918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:25.830918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:25.830918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:25.830918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.831138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:25.831138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:25.831138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:25.831138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:25.831138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:25.831138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:25.831138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:25.831138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:25.831138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:25.831138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:25.831138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:25.831138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:25.831138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:25.831413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.831413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:25.831827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.831827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:25.831827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:25.831827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:25.831827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:25.831827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:25.831827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.831827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:25.831827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:25.831827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.831827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:25.831827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:25.831827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.831827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:25.831827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:25.831827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.832737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:25.832737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:25.832737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.832737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:25.832737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:25.832737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:25.832737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.832737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:25.832737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:25.832737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:25.832737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:25.832737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:25.832737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:25.832737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:25.832737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:25.832737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:25.832737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:25.832737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:25.832737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:25.833345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:25.833345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.833345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:25.838657 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:25.838836 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:25.838943 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:25.839039 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:36:25.841727 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:25.841727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.841727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:25.841727 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:25.841727 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:25.841727 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:25.841727 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:25.841727 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:25.841727 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.841727 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:25.841727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.841727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:25.841727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:25.841727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:25.842020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.842020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:25.842020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:25.842020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:25.842020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.842020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:25.842020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:25.842020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:25.842020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:25.842020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:25.842020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:25.842020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:25.842020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:25.842020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:25.842020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:25.842020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:25.842020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:25.842020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:25.842322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.842322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:25.842322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.842322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:25.842322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:25.842322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:25.842322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.842322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:25.842322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:25.842322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:36:25.842322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:25.842322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:25.842541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.842541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:25.842541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:25.842541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:25.842541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.842541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:25.842541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:25.842541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:25.842541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:25.842541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:25.842541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:25.842541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:25.842541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:25.842541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:25.842773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:25.842773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:25.842773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:25.842773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:25.842773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.842773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:25.842773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.842773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:25.842773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:25.842773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:25.842773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:25.842773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:25.842773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.842773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:25.842998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:25.842998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.842998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:25.842998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:25.842998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.842998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:25.842998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:25.842998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.842998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:25.842998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:25.842998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:36:25.842998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:25.842998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:25.842998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.842998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:25.842998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:25.842998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:25.842998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.842998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:25.843294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:25.843294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:25.843294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:25.843294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:25.843294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:25.843294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:25.843294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:25.843294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:25.843294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:25.843294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:25.843294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:25.843294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:25.843294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.843294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:25.846730 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:25.846924 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:25.847036 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:25.847146 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:25.847524 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:25.847787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.848220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:25.848220 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:25.848220 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:25.848220 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:25.848220 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:25.848220 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:25.848220 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.848220 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:25.848788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.848788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:25.848788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:25.848788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:25.848788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.848788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:25.848788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:25.848788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:25.848788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.848788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:25.848788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:25.848788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:25.848788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:25.848788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:25.848788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:25.849021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:25.849021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:25.849021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:25.849021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:25.849021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:25.849021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:25.849021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:25.849021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.849021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:25.849420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.849420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:25.849420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:25.849420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:25.849420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.849420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:25.849420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:25.849420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:25.849420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:25.849420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.849420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:25.849643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:25.849643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:25.849643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.849643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:25.849643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:25.849643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:25.849643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:25.849643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:25.849643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:25.849643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:25.849643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:25.849643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:25.849643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:25.849643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:25.849842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:25.849842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:25.849842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.849842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:25.850186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.850186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:25.850186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:25.850186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:25.850186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:25.850186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:25.850186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.850186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:25.850186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:25.850186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.850186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:25.850186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:25.850186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.850186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:25.850186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:25.850186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.850186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:25.850659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:25.850659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:25.850659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:25.850659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.850659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:25.850659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:25.850659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:25.850659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.850659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:25.850659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:25.850659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:25.850659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:25.850659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:25.850659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:25.850659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:25.850659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:25.850659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:25.850659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:25.850659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:25.850931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:25.850931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:25.850931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.850931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:25.872028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:25.872299 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:25.872389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:25.872537 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:25.872844 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:25.873046 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:25.873223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.873683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:25.873683 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:25.873683 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:25.873683 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:25.873683 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:25.873683 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:25.873683 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.873683 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:25.874248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.874248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:25.874248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:25.874248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:25.874248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.874248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:25.874248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:25.874248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.874248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:25.874248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:25.874248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:25.874248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.874496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:25.874496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:25.874496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:25.874496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:25.874496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:25.874496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:25.874496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:25.874496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:25.874496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:25.874496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:25.874496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:25.874496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:25.874496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:25.874727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.874727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:25.875151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.875151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:25.875151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:25.875151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:25.875151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:25.875151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:25.875151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.875151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:25.875151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:25.875151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.875151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:25.875151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:25.875151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.875151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:25.875151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:25.875151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:25.875764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:25.875764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:25.875764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.875764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:25.875764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:25.875764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:25.875764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:25.875764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:25.875764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:25.875764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:25.875764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:25.875764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:25.875764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:25.875764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:25.875764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:25.875764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:25.875764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:25.875764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:25.875764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:25.876114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:25.876114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:25.876114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:28.746807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:28.747624 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:28.747724 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:28.748344 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:28.777419 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:28.777659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:28.778139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:28.778139 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:28.778139 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:28.778139 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:28.778139 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:28.778139 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:28.778139 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:28.778139 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:28.778879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:28.778879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:28.778879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:28.778879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:28.778879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:28.778879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:28.778879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:28.778879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:28.778879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:28.778879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:28.778879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:28.778879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:28.779544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:28.779544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:28.779544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:28.779544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:28.779544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:28.779544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:28.779544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:28.779544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:28.779544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:28.779544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:28.779544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:28.779544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:28.779544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:28.779930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:28.779930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:28.779930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:28.779930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:28.779930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:28.779930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:28.779930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:28.779930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:28.779930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:28.779930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:28.779930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:28.779930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:28.779930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:28.779930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:28.779930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:28.780176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:28.780176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:28.780176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:28.780176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:28.780176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:28.780176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:28.780176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:28.780176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:28.780176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:28.780176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:28.780176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:28.780176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:28.780176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:28.780176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:28.780176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:28.780176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:28.780176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:28.780176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:28.780176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:28.780509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:28.780509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:28.780509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:28.780509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:28.780509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:28.780509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:30.828238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:30.829010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:30.829205 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:30.830131 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:30.830400 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:30.830620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:30.830779 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:30.830925 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:30.831074 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:30.831243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.831956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:30.831956 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:30.831956 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:30.831956 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:30.831956 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:30.831956 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:30.831956 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.831956 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:30.832952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.832952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:30.832952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:30.832952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:30.832952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.832952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:30.832952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:30.832952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.832952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:30.832952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:30.832952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:30.832952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.833338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:30.833338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:30.833338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:30.833338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:30.833338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:30.833338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:30.833338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:30.833338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:30.833338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:30.833338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:30.833338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:30.833338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:30.833338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:30.833562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.833562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:30.834743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.834743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:30.834743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:30.834743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:30.834743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:30.834743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:30.834743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.834743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:30.834743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:30.834743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.834743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:30.834743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:30.834743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.834743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:30.834743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:30.834743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.835396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:30.835396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:30.835396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.835396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:30.835396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:30.835396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:30.835396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.835396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:30.835396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:30.835396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:30.835396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:30.835396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:30.835396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:30.835396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:30.835396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:30.835396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:30.835396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:30.835396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:30.835396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:30.835767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:30.835767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.835767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:30.838988 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:30.839426 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:30.839585 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:30.839731 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:36:30.840128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:30.840417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.841157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:30.841157 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:30.841157 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:30.841157 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:30.841157 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:30.841157 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:30.841157 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.841157 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:30.842291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.842291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:30.842291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:30.842291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:30.842291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.842291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:30.842291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:30.842291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:30.842291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.842291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:30.842291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:30.842291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:30.842291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:30.842291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:30.842291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:30.842807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:30.842807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:30.842807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:30.842807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:30.842807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:30.842807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:30.842807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:30.842807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.842807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:30.844620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.844620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:30.844620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:30.844620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:30.844620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.844620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:30.844620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:30.844620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:36:30.844620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:30.844620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:30.844620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.844620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:30.845012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:30.845012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:30.845012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.845012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:30.845012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:30.845012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:30.845012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:30.845012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:30.845012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:30.845012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:30.845012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:30.845012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:30.845012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:30.845012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:30.845387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:30.845387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:30.845387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.845387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:30.845387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.845387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:30.845387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:30.845387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:30.845387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:30.845387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:30.845387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.845387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:30.845387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:30.845387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.845783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:30.845783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:30.845783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.845783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:30.845783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:30.845783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.845783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:30.845783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:30.845783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:36:30.845783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:30.845783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:30.845783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.845783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:30.845783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:30.845783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:30.845783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.845783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:30.845783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:30.845783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:30.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:30.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:30.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:30.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:30.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:30.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:30.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:30.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:30.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:30.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:30.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.846148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:30.847805 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:30.847805 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:30.847805 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:30.847805 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:30.847805 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:30.847805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.848261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:30.848261 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:30.848261 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:30.848261 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:30.848261 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:30.848261 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:30.848261 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.848261 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:30.848975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.848975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:30.848975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:30.848975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:30.848975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.848975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:30.848975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:30.848975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:30.848975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.848975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:30.848975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:30.848975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:30.848975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:30.848975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:30.848975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:30.849819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:30.849819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:30.849819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:30.849819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:30.849819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:30.849819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:30.849819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:30.849819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.849819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:30.850228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.850228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:30.850228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:30.850228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:30.850228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.850228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:30.850228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:30.850228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:30.850228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:30.850228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.850228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:30.850548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:30.850548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:30.850548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.850548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:30.850548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:30.850548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:30.850548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:30.850548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:30.850548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:30.850548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:30.850548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:30.850548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:30.850548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:30.850548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:30.850892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:30.850892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:30.850892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.850892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:30.851290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.851290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:30.851290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:30.851290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:30.851290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:30.851290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:30.851290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.851290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:30.851290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:30.851290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.851290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:30.851290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:30.851290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.851290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:30.851290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:30.851290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.851290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:30.851775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:30.851775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:30.851775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:30.851775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.851775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:30.851775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:30.851775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:30.851775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.851775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:30.851775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:30.851775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:30.851775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:30.851775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:30.851775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:30.851775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:30.851775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:30.851775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:30.851775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:30.851775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:30.852338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:30.852338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:30.852338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.852338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:30.872336 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:30.872569 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:30.872685 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:30.872821 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:30.873242 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:30.873514 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:30.873752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.874193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:30.874193 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:30.874193 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:30.874193 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:30.874193 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:30.874193 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:30.874193 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.874193 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:30.874920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.874920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:30.874920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:30.874920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:30.874920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.874920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:30.874920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:30.874920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.874920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:30.874920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:30.874920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:30.874920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.875126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:30.875126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:30.875126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:30.875126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:30.875126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:30.875126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:30.875126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:30.875126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:30.875126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:30.875126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:30.875126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:30.875126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:30.875126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:30.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:30.876026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.876026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:30.876026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:30.876026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:30.876026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:30.876026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:30.876026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.876026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:30.876026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:30.876026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.876026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:30.876026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:30.876026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.876026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:30.876026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:30.876026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:30.876317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:30.876317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:30.876317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.876317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:30.876317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:30.876317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:30.876317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:30.876317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:30.876317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:30.876317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:30.876317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:30.876317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:30.876317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:30.876317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:30.876317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:30.876317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:30.876317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:30.876317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:30.876317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:30.876752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:30.876752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:30.876752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:33.747792 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:33.748353 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:33.748353 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:33.748918 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:33.780483 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:33.780668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:33.781085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:33.781085 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:33.781085 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:33.781085 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:33.781085 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:33.781085 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:33.781085 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:33.781085 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:33.781802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:33.781802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:33.781802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:33.781802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:33.781802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:33.781802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:33.781802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:33.781802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:33.781802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:33.781802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:33.781802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:33.781802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:33.782105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:33.782105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:33.782105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:33.782105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:33.782105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:33.782105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:33.782105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:33.782105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:33.782105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:33.782105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:33.782105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:33.782105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:33.782105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:33.782396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:33.782396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:33.782671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:33.782671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:33.782671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:33.782671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:33.782671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:33.782671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:33.782671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:33.782671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:33.782671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:33.782671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:33.782671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:33.782671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:33.782671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:33.782671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:33.782671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:33.782671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:33.783031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:33.783031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:33.783031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:33.783031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:33.783031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:33.783031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:33.783031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:33.783031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:33.783031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:33.783031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:33.783031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:33.783031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:33.783031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:33.783031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:33.783031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:33.783031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:33.783031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:33.783031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:33.783031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:33.783475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:33.783475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:33.783475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:35.829176 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:35.830764 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:35.830764 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:35.830764 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:35.831014 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:35.831102 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:35.831245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:35.831442 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:35.832526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:35.832526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-39193d81-b808-484f-b34b-96ed9de26740, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:35.832526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.832526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:35.832526 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:35.832526 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:35.832526 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:35.832526 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:35.832526 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:35.832526 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.832526 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:35.833406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.833406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:35.833406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:35.833406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:35.833406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.833406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:35.833406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:35.833406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.833406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:35.833406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:35.833406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:35.833406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.834725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:35.834725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:35.834725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:35.834725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:35.834725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:35.834725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:35.834725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:35.834725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:35.834725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:35.834725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:35.834725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:35.834725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:35.834725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:35.834944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.834944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:35.834944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.834944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:35.834944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:35.834944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:35.834944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:35.834944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:35.834944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.834944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:35.834944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:35.834944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.834944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:35.834944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:35.834944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.835208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:35.835208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:35.835208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.835208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:35.835208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:35.835208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.835208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:35.835208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:35.835208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:35.835208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.835208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:35.835208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:35.835208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:35.835208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:35.835208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:35.835208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:35.835208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:35.835208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:35.835208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:35.835537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:35.835537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:35.835537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:35.835537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:35.835537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.835537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:35.839759 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:35.839759 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:35.839759 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:35.839894 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:36:35.840962 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:35.840962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.840962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:35.840962 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:35.840962 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:35.840962 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:35.840962 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:35.840962 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:35.840962 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.840962 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:35.841331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.841331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:35.841331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:35.841331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:35.841331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.841331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:35.841331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:35.841331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:35.841331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.841331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:35.841331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:35.841331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:35.841331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:35.841331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:35.841331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:35.841616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:35.841616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:35.841616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:35.841616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:35.841616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:35.841616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:35.841616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:35.841616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.841616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:35.841780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.841780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:35.841780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:35.841780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:35.841780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.841780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:35.841780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:35.841780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:36:35.841780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:35.841780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:35.841780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.841780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:35.841991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:35.841991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:35.841991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.841991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:35.841991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:35.841991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:35.841991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:35.841991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:35.841991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:35.841991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:35.841991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:35.841991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:35.841991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:35.841991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:35.842210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:35.842210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:35.842210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.842210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:35.842871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.842871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:35.842871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:35.842871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:35.842871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:35.842871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:35.842871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.842871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:35.842871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:35.842871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.842871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:35.842871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:35.842871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.842871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:35.842871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:35.842871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.843936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:35.843936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:35.843936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:36:35.843936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:35.843936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:35.843936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.843936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:35.843936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:35.843936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:35.843936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.843936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:35.843936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:35.843936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:35.843936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:35.843936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:35.843936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:35.843936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:35.843936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:35.846442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:35.846442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:35.846442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:35.846442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:35.846442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:35.846442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.846442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:35.847239 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:35.847472 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:35.847557 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:35.847653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:35.847929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:35.848523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.848927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:35.848927 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:35.848927 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:35.848927 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:35.848927 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:35.848927 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:35.848927 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.848927 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:35.849487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.849487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:35.849487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:35.849487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:35.849487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.849487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:35.849487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:35.849487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:35.849487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.849487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:35.849487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:35.849487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:35.849487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:35.849487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:35.849487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:35.850474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:35.850474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:35.850474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:35.850474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:35.850474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:35.850474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:35.850474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:35.850474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.850474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:35.850474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.850474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:35.850474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:35.850474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:35.850474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.850897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:35.850897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:35.850897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:35.850897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:35.850897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.850897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:35.850897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:35.850897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:35.850897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.850897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:35.850897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:35.850897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:35.850897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:35.851210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:35.851210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:35.851210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:35.851210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:35.851210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:35.851210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:35.851210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:35.851210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:35.851210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:35.851210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.851210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:35.851210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.851549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:35.851549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:35.851549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:35.851549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:35.851549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:35.851549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.851549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:35.851549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:35.851549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.851549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:35.851549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:35.851549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.851549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:35.851549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:35.851549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.851549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:35.851549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:35.851549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:35.851549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.852486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:35.872854 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:35.873247 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:35.876010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:35.876010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:35.876010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:35.876010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:35.876010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:35.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:35.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:35.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:35.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:35.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:35.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:35.877146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.877146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:35.877146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:35.877146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:35.877146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.877146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:35.877146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:35.877146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.877146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:35.877146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:35.877146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:35.877146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.877427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:35.877427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:35.877427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:35.877427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:35.877427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:35.877427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:35.877427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:35.877427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:35.877427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:35.877427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:35.877427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:35.877427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:35.877427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:35.877673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.877673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:35.877901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.877901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:35.877901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:35.877901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:35.877901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:35.877901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:35.877901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.877901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:35.877901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:35.877901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.877901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:35.877901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:35.877901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.877901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:35.877901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:35.877901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:35.878293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:35.878293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:35.878293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.878293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:35.878293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:35.878293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:35.878293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:35.878293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:35.878293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:35.878293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:35.878293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:35.878293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:35.878293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:35.878293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:35.878293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:35.878293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:35.878293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:35.878293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:35.878293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:35.878642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:35.878642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:35.878642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:38.748270 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:38.756130 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:38.756130 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:38.756130 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:38.795663 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:38.795972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:38.796186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:38.796186 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:38.796186 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:38.796186 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:38.796186 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:38.796186 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:38.796186 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:38.796186 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:38.797499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:38.797499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:38.797499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:38.797499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:38.797499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:38.797499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:38.797499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:38.797499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:38.797499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:38.797499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:38.797499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:38.797499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:38.798536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:38.798536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:38.798536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:38.798536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:38.798536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:38.798536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:38.798536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:38.798536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:38.798536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:38.798536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:38.798536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:38.798536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:38.798536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:38.800235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:38.800235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:38.800235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:38.800235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:38.800235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:38.800235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:38.800235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:38.800235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:38.800235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:38.800235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:38.800235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:38.800235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:38.800235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:38.800235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:38.800235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:38.801296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:38.801296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:38.801296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:38.801296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:38.801296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:38.801296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:38.801296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:38.801296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:38.801296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:38.801296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:38.801296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:38.801296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:38.801296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:38.801296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:38.801296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:38.801296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:38.801296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:38.801296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:38.801296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:38.802907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:38.802907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:38.802907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:38.802907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:38.802907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:38.802907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:40.831622 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:40.836302 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:40.836302 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:40.836302 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:40.836302 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:40.836302 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:40.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:40.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:40.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:40.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-39193d81-b808-484f-b34b-96ed9de26740, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:40.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-840c8102-5b85-4c5a-b864-0d827bd91136, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:40.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.841130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:40.841130 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:40.841130 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:40.841130 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:40.841130 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:40.841130 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:40.841130 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.841130 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:40.841130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.841130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:40.841130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:40.841130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:40.841130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.841130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:40.841130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:40.844569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.844569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:40.844569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:40.844569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:40.844569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.844569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:40.844569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:40.844569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:40.844569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:40.844569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:40.844569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:40.844569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:40.844569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:40.844569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:40.845325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:40.845325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:40.845325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:40.845325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:40.845325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.845325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:40.845325 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:40.845325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.845325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:40.845325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:40.845325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:40.845325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:40.845854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:40.848555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:40.848555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:40.848555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:40.848555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:40.848555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:40.848555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:40.848555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:40.848555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:40.848555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:40.848555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:40.848555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:40.848555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.848555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:40.848555 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:40.848555 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:40.849486 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:36:40.849486 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:40.849486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.849486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:40.849486 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:40.849486 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:40.849486 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:40.849486 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:40.849486 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:40.849486 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.849486 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:40.849486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.849486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:40.850412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:40.850412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:40.850412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.850412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:40.850412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:40.850412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:40.850412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.850412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:40.850412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:40.850412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:40.850412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:40.850412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:40.850412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:40.850412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:40.850412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:40.850412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:40.850412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:40.850764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:40.850764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:40.850764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:40.850764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.850764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:40.850764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.850764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:40.850764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:40.850764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:40.850764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.850764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:40.850764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:40.850764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:36:40.851502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:40.851502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:40.851502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.851502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:40.851502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:40.851502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:40.851502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.851502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:40.851502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:40.851502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:40.851502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:40.851502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:40.851502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:40.851717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:40.851717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:40.851717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:40.851717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:40.851717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:40.851717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:40.851717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:40.851717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.851717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:40.851717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.851717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:40.851717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:40.852531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:40.852531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:40.852531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.852531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:40.852531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:40.852531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:40.852531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:40.852531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:40.852531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:40.852531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:40.852531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:40.852531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:40.852531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:40.852531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:40.852531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:40.852531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:40.852531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.852531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:40.852858 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:40.852858 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:40.852858 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:40.852858 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:40.852858 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:40.852858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.853073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:40.853073 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:40.853073 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:40.853073 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:40.853073 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:40.853073 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:40.853073 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.853073 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:40.853073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.853073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:40.853073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:40.853073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:40.853073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.853073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:40.853073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:40.853073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:40.853073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.853073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:40.853409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:40.853409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:40.853409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:40.853409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:40.853409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:40.853409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:40.853409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:40.853409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:40.853409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:40.853409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:40.853409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:40.853409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:40.853409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.853409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:40.853409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.854268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:40.854268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:40.854268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:40.854268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.854268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:40.854268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:40.854268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:40.854268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:40.854268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.854268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:40.854268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:40.854268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:40.854268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.854549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:40.854549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:40.854549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:40.854549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:40.854549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:40.854549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:40.854549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:40.854549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:40.854549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:40.854549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:40.854549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:40.854549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:40.854549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:40.854853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.854853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:40.854853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.854853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:40.854853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:40.854853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:40.854853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:40.854853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:40.854853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.854853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:40.854853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:40.854853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.854853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:40.854853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:40.854853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.855138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:40.855138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:40.855138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.855138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:40.855138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:40.855138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:40.855138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:40.855138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.855138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:40.855138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:40.855138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:40.855138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.855138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:40.855138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:40.855138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:40.855138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:40.855138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:40.855138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:40.855138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:40.855492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:40.855492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:40.855492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:40.855492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:40.855492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:40.855492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:40.855492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.855492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:40.873383 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:40.873785 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:40.873908 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:40.874081 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:40.875568 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:40.875768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:40.875968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.876682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:40.876682 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:40.876682 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:40.876682 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:40.876682 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:40.876682 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:40.876682 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.876682 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:40.877352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.877352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:40.877352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:40.877352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:40.877352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.877352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:40.877352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:40.877352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.877352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:40.877352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:40.877352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:40.877352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.878033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:40.878033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:40.878033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:40.878033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:40.878033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:40.878033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:40.878033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:40.878033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:40.878033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:40.878033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:40.878033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:40.878033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:40.878033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:40.878226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.878226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:40.878226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.878226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:40.878226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:40.878226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:40.878226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:40.878226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:40.878226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.878226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:40.878226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:40.878226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.878226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:40.878226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:40.878226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.878505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:40.878505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:40.878505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:40.878505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:40.878505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:40.878505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.878505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:40.878505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:40.878505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:40.878505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:40.878505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:40.878505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:40.878505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:40.878505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:40.878505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:40.878505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:40.878505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:40.878505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:40.878505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:40.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:40.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:40.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:40.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:40.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:40.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:43.750332 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:43.755190 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:43.755190 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:43.755190 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:43.787137 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:43.787464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:43.789234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:43.789234 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:43.789234 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:43.789234 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:43.789234 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:43.789234 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:43.789234 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:43.789234 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:43.789234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:43.789234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:43.789234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:43.789234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:43.789234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:43.789234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:43.789234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:43.792900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:43.792900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:43.792900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:43.792900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:43.792900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:43.792900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:43.792900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:43.792900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:43.792900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:43.792900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:43.792900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:43.792900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:43.792900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:43.792900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:43.794104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:43.794104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:43.794104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:43.794104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:43.794104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:43.794104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:43.794104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:43.794104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:43.794104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:43.794104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:43.794104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:43.794104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:43.794104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:43.794104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:43.794713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:43.795962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:43.795962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:43.795962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:43.795962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:43.795962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:43.795962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:43.795962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:43.795962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:43.795962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:45.832136 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:45.833350 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:45.833350 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:45.833728 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:45.833908 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:45.834598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:45.834598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:45.834598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:45.834598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:45.834598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-39193d81-b808-484f-b34b-96ed9de26740, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:45.836468 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-840c8102-5b85-4c5a-b864-0d827bd91136, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:45.836468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.836468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:45.836468 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:45.836468 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:45.836468 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:45.836468 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:45.836468 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:45.836468 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.836468 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:45.836468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.836468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:45.836468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:45.836468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:45.837139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.837139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:45.837139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:45.837139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.837139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:45.837139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:45.837139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:45.837139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.837139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:45.837139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:45.837139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:45.837139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:45.837139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:45.837139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:45.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:45.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:45.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:45.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:45.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:45.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:45.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:45.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:45.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:45.837392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:45.837673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.838128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:45.838128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:45.838128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:45.838128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:45.838128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:45.838128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:45.838128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:45.838128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:45.838128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:45.838128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:45.838128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:45.838128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:45.838128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:45.838128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.838128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:45.842132 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:45.842519 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:45.842686 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:45.842804 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:36:45.843139 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:45.843321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.843791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:45.843791 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:45.843791 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:45.843791 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:45.843791 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:45.843791 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:45.843791 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.843791 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:45.844669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.844669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:45.844669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:45.844669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:45.844669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.844669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:45.844669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:45.844669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:45.844669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.844669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:45.844669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:45.844669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:45.844669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:45.844669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:45.844669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:45.846127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:45.846127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:45.846127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:45.846127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:45.846127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:45.846127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:45.846127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:45.846127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.846127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:45.846127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.846127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:45.846127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:45.846127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:45.846127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.846867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:45.846867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:45.846867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:36:45.846867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:45.846867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:45.846867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.846867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:45.846867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:45.846867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:45.846867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.846867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:45.846867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:45.846867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:45.847099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:45.847099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:45.847099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:45.847099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:45.847099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:45.847099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:45.847099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:45.847099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:45.847099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:45.847099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:45.847099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.847099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:45.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:45.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:45.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:45.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:45.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:45.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:45.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:45.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:45.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:45.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:45.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:45.847423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.847861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:45.847861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:45.847861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:36:45.847861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:45.847861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:45.847861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.847861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:45.847861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:45.847861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:45.847861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.847861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:45.847861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:45.847861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:45.847861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:45.847861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:45.847861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:45.847861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:45.847861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:45.848286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:45.848286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:45.848286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:45.848286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:45.848286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:45.848286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.848286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:45.848620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:45.848889 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:45.849021 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:45.849165 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:45.849450 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:45.849620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.850095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:45.850095 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:45.850095 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:45.850095 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:45.850095 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:45.850095 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:45.850095 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.850095 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:45.850907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.850907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:45.850907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:45.850907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:45.850907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.850907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:45.850907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:45.850907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:45.850907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.850907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:45.850907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:45.850907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:45.850907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:45.850907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:45.850907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:45.851196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:45.851196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:45.851196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:45.851196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:45.851196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:45.851196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:45.851196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:45.851196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.851196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:45.851819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.851819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:45.851819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:45.851819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:45.851819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.851819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:45.851819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:45.851819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:45.851819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:45.851819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.851819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:45.852109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:45.852109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:45.852109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.852109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:45.852109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:45.852109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:45.852109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:45.852109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:45.852109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:45.852109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:45.852109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:45.852109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:45.852109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:45.852109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:45.852350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:45.852350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:45.852350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.852350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:45.852552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.852552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:45.852552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:45.852552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:45.852552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:45.852552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:45.852552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.852552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:45.852552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:45.852552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.852552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:45.852552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:45.852552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.852552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:45.852552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:45.852552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.852552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:45.852868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:45.852868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:45.852868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:45.852868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.852868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:45.852868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:45.852868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:45.852868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.852868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:45.852868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:45.852868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:45.852868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:45.852868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:45.852868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:45.852868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:45.852868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:45.852868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:45.852868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:45.852868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:45.853152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:45.853152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:45.853152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.853152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:45.873758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:45.874137 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:45.874266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:45.874459 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:45.874923 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:45.875112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:45.875362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.875807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:45.875807 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:45.875807 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:45.875807 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:45.875807 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:45.875807 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:45.875807 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.875807 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:45.876545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.876545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:45.876545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:45.876545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:45.876545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.876545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:45.876545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:45.876545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.876545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:45.876545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:45.876545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:45.876545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.877017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:45.877017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:45.877017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:45.877017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:45.877017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:45.877017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:45.877017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:45.877017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:45.877017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:45.877017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:45.877017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:45.877017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:45.877017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:45.877215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.877215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:45.877355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.877355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:45.877355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:45.877355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:45.877355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:45.877355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:45.877355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.877355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:45.877355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:45.877355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.877355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:45.877355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:45.877355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.877355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:45.877355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:45.877355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:45.877653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:45.877653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:45.877653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.877653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:45.877653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:45.877653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:45.877653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:45.877653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:45.877653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:45.877653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:45.877653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:45.877653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:45.877653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:45.877653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:45.877653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:45.877653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:45.877653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:45.877653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:45.877653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:45.877962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:45.877962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:45.877962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:48.750906 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:48.751475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:48.751475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:48.751905 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:48.806331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:48.806520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:48.807374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:48.807374 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:48.807374 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:48.807374 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:48.807374 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:48.807374 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:48.807374 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:48.807374 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:48.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.057 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:48.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:48.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:48.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:48.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:48.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:48.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:48.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:48.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:48.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:48.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:48.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:48.809611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:48.809611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:48.809611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:48.809611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:48.809611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:48.809611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:48.809611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:48.809611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:48.809611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:48.809611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:48.809611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:48.809611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:48.809611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:48.809817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:48.809817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:48.810386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:48.810386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:48.810386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:48.810386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:48.810386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:48.810386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:48.810386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:48.810386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:48.810386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:48.810386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:48.810386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:48.810386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:48.810386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:48.810386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:48.810386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:48.810386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:48.810733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:48.810733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:48.810733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:48.810733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:48.810733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:48.810733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:48.810733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:48.810733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:48.810733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:48.810733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:48.810733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:48.810733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:48.810733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:48.810733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:48.810733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:48.810733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:48.810733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:48.810733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:48.810733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:48.811124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:48.811124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:48.811124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:50.832567 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:50.833041 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:50.833041 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:50.833793 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:50.833996 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:50.834116 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:50.834234 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:50.834368 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:50.834466 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:50.834831 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-840c8102-5b85-4c5a-b864-0d827bd91136, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:50.834957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.835427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:50.835427 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:50.835427 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:50.835427 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:50.835427 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:50.835427 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:50.835427 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.835427 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:50.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:50.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:50.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:50.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:50.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:50.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:50.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:50.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:50.836176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.836467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:50.836467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:50.836467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:50.836467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:50.836467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:50.836467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:50.836467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:50.836467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:50.836467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:50.836467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:50.836467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:50.836467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:50.836467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:50.836699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.836699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:50.836947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.836947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:50.836947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:50.836947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:50.836947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:50.836947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:50.836947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.836947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:50.836947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:50.836947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.836947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:50.836947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:50.836947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.836947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:50.836947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:50.836947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.837211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:50.837211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:50.837211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.837211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:50.837211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:50.837211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:50.837211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.837211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:50.837211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:50.837211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:50.837211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:50.837211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:50.837211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:50.837211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:50.837211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:50.837211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:50.837211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:50.837211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:50.837211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:50.837521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:50.837521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.837521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:50.842518 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:50.842684 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:50.842792 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:50.842887 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:36:50.843183 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:50.843373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.843839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:50.843839 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:50.843839 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:50.843839 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:50.843839 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:50.843839 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:50.843839 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.843839 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:50.844419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.844419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:50.844419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:50.844419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:50.844419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.844419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:50.844419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:50.844419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:50.844419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.844419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:50.844419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:50.844419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:50.844419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:50.844419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:50.844419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:50.844686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:50.844686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:50.844686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:50.844686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:50.844686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:50.844686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:50.844686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:50.844686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.844686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:50.845075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.845075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:50.845075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:50.845075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:50.845075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.845075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:50.845075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:50.845075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:36:50.845075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:50.845075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:50.845075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.845075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:50.845324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:50.845324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:50.845324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.845324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:50.845324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:50.845324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:50.845324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:50.845324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:50.845324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:50.845324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:50.845324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:50.845324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:50.845324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:50.845324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:50.845554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:50.845554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:50.845554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.845554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:50.845944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.845944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:50.845944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:50.845944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:50.845944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:50.845944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:50.845944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.845944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:50.845944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:50.845944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.845944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:50.845944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:50.845944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.845944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:50.845944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:50.845944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.846209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:50.846209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:50.846209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:36:50.846209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:50.846209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:50.846209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.846209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:50.846209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:50.846209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:50.846209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.846209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:50.846209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:50.846209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:50.846209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:50.846209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:50.846209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:50.846209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:50.846209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:50.846511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:50.846511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:50.846511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:50.846511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:50.846511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:50.846511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.846511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:50.848880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:50.849062 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:50.849177 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:50.849315 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:50.849591 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:50.849775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.850118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:50.850118 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:50.850118 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:50.850118 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:50.850118 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:50.850118 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:50.850118 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.850118 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:50.850695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.850695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:50.850695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:50.850695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:50.850695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.850695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:50.850695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:50.850695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:50.850695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.850695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:50.850695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:50.850695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:50.850695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:50.850695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:50.850695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:50.851065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:50.851065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:50.851065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:50.851065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:50.851065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:50.851065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:50.851065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:50.851065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.851065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:50.851328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.851328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:50.851328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:50.851328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:50.851328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.851328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:50.851328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:50.851328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:50.851328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:50.851328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.851328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:50.852001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:50.852001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:50.852001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.852001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:50.852001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:50.852001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:50.852001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:50.852001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:50.852001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:50.852001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:50.852001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:50.852001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:50.852001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:50.852001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:50.852302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:50.852302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:50.852302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.852302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:50.852302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.852302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:50.852302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:50.852302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:50.852302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:50.852302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:50.852302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.852302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:50.852302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:50.852302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.852616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:50.852616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:50.852616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.852616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:50.852616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:50.852616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.852616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:50.852616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:50.852616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:50.852616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:50.852616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.852616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:50.852616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:50.852616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:50.852616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.852616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:50.852616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:50.852616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:50.852616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:50.852975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:50.852975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:50.852975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:50.852975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:50.852975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:50.852975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:50.852975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:50.852975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:50.852975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:50.852975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.852975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:50.874133 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:50.874356 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:50.874447 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:50.874585 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:50.874881 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:50.875068 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:50.875246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.875658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:50.875658 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:50.875658 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:50.875658 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:50.875658 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:50.875658 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:50.875658 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.875658 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:50.876422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.876422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:50.876422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:50.876422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:50.876422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.876422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:50.876422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:50.876422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.876422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:50.876422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:50.876422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:50.876422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.877014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:50.877014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:50.877014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:50.877014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:50.877014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:50.877014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:50.877014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:50.877014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:50.877014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:50.877014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:50.877014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:50.877014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:50.877014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:50.877248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.877248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:50.877248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.877248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:50.877248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:50.877248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:50.877248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:50.877248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:50.877248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.877248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:50.877248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:50.877248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.877248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:50.877248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:50.877248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.877512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:50.877512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:50.877512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:50.877512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:50.877512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:50.877512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.877512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:50.877512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:50.877512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:50.877512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:50.877512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:50.877512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:50.877512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:50.877512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:50.877512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:50.877512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:50.877512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:50.877512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:50.877512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:50.877877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:50.877877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:50.877877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:50.877877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:50.877877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:50.877877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:53.751212 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:53.751655 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:53.751655 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:53.752220 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:53.757103 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=564,neutron:DELETE=16 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:36:53.786207 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:53.786429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:53.786957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:53.786957 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:53.786957 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:53.786957 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:53.786957 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:53.786957 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:53.786957 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:53.786957 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:53.787733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:53.787733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:53.787733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:53.787733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:53.787733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:53.787733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:53.787733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:53.787733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:53.787733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:53.787733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:53.787733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:53.787733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:53.788039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:53.788039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:53.788039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:53.788039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:53.788039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:53.788039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:53.788039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:53.788039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:53.788039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:53.788039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:53.788039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:53.788039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:53.788039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:53.788338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:53.788338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:53.789543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:53.789543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:53.789543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:53.789543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:53.789543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:53.789543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:53.789543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:53.789543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:53.789543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:53.789543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:53.789543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:53.789543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:53.789543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:53.789543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:53.789543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:53.789543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:53.789903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:53.789903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:53.789903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:53.789903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:53.789903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:53.789903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:53.789903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:53.789903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:53.789903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:53.789903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:53.789903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:53.789903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:53.789903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:53.789903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:53.789903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:53.789903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:53.789903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:53.789903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:53.789903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:53.790272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:53.790272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:53.790272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:55.832935 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:55.833386 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:55.833439 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:55.834300 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:55.834433 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:55.834598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:55.834752 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:55.835187 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:55.835187 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:55.835187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.836807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:55.836807 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:55.836807 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:55.836807 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:55.836807 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:55.836807 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:55.836807 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.836807 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:55.836807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.836807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:55.836807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:55.836807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:55.836807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.836807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:55.836807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:55.837732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.837732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:55.837732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:55.837732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:55.837732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.837732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:55.837732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:55.837732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:55.837732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:55.837732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:55.837732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:55.837732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:55.837732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:55.837732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:55.838072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:55.838072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:55.838072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:55.838072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:55.838072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.838072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:55.838072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.838072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:55.838072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:55.838072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:55.838072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:55.838072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:55.838072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.838072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:55.838389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:55.838748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:55.838748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:55.838748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:55.838748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:55.838748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:55.838748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:55.838748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:55.838748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:55.838748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.838748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:55.842766 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:55.842957 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:55.843080 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:55.843225 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:36:55.843530 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:55.843723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.844133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:55.844133 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:55.844133 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:55.844133 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:55.844133 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:55.844133 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:55.844133 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.844133 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:55.844918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.844918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:55.844918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:55.844918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:55.844918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.844918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:55.844918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:55.844918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:55.844918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.844918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:55.844918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:55.844918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:55.844918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:55.844918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:55.844918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:55.845198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:55.845198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:55.845198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:55.845198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:55.845198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:55.845198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:55.845198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:55.845198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.845198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:55.845843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.845843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:55.845843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:55.845843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:55.845843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.845843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:55.845843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:55.845843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:36:55.845843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:55.845843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:55.845843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.845843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:55.846058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:55.846058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:55.846058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.846058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:55.846058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:55.846058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:55.846058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:55.846058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:55.846058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:55.846058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:55.846058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:55.846058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:55.846058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:55.846058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:55.846363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:55.846363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:55.846363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.846363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:55.846987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.846987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:55.846987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:55.846987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:55.846987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:55.846987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:55.846987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.846987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:55.846987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:55.846987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.846987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:55.846987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:55.846987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.846987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:55.846987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:55.846987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.847314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:36:55.847314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:36:55.847314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:36:55.847314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:55.847314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:55.847314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.847314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:55.847314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:55.847314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:55.847314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.847314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:55.847314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:55.847314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:55.847314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:55.847314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:55.847314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:55.847314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:55.847314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:55.847604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:55.847604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:55.847604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:55.847604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:55.847604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:55.847604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.847604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:55.849146 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:55.849330 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:55.849463 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:55.849622 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:55.849888 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:55.850084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.850489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:55.850489 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:55.850489 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:55.850489 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:55.850489 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:55.850489 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:55.850489 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.850489 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:55.851268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.851268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:36:55.851268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:55.851268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:55.851268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.851268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:55.851268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:55.851268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:36:55.851268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.851268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:36:55.851268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:55.851268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:36:55.851268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:55.851268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:36:55.851268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:36:55.851525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:55.851525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:36:55.851525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:55.851525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:36:55.851525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:36:55.851525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:55.851525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:36:55.851525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.851525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:36:55.852263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.852263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:55.852263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:55.852263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:55.852263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.852263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:55.852263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:55.852263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:55.852263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:55.852263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.852263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:55.852520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:55.852520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:55.852520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.852520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:55.852520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:55.852520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:55.852520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:55.852520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:55.852520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:55.852520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:55.852520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:55.852520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:55.852520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:55.852520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:55.852756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:55.852756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:55.852756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.852756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:55.853491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.853491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:55.853491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:55.853491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:55.853491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:55.853491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:55.853491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.853491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:55.853491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:55.853491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.853491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:55.853491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:55.853491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.853491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:55.853491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:55.853491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.853491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:36:55.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:36:55.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:36:55.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:36:55.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:55.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:55.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:55.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:55.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:55.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:55.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:55.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:55.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:55.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:55.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:55.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:55.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:55.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:55.854248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:55.854248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:55.854248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.854248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:55.874636 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:55.875044 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:55.875175 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:55.875372 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:55.875727 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:55.875986 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:55.876229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.876780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:55.876780 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:55.876780 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:55.876780 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:55.876780 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:55.876780 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:55.876780 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.876780 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:55.877373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.877373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:55.877373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:55.877373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:55.877373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.877373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:55.877373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:55.877373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.877373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:55.877373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:55.877373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:55.877373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.877609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:55.877609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:55.877609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:55.877609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:55.877609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:55.877609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:55.877609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:55.877609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:55.877609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:55.877609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:55.877609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:55.877609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:55.877609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:55.877803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.877803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:55.878038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.878038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:55.878038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:55.878038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:55.878038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:55.878038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:55.878038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.878038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:55.878038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:55.878038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.878038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:55.878038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:55.878038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.878038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:55.878038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:55.878038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:55.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:36:55.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:55.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:55.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:55.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:55.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:55.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:55.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:55.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:55.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:55.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:55.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:55.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:55.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:55.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:55.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:55.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:55.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:55.878749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:55.878749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:55.878749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:36:58.751621 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:36:58.752076 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:36:58.752076 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:36:58.752555 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:36:58.781414 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:36:58.781676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:58.782267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:36:58.782267 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:58.782267 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:36:58.782267 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:36:58.782267 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:58.782267 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:36:58.782267 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:58.782267 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:36:58.783279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:58.783279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:36:58.783279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:58.783279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:36:58.783279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:58.783279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:58.783279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:58.783279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:58.783279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:58.783279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:58.783279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:36:58.783279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:58.784108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:36:58.784108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:58.784108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:36:58.784108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:58.784108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:36:58.784108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:36:58.784108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:58.784108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:36:58.784108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:58.784108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:36:58.784108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:36:58.784108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:58.784108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:36:58.784359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:58.784359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:36:58.784359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:58.784359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:36:58.784359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:36:58.784359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:36:58.784359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:36:58.784359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:36:58.784359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:58.784359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:36:58.784359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:36:58.784359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:58.784359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:36:58.784359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:36:58.784359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:36:58.784620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:36:58.784620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:36:58.784620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:36:58.784620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:36:58.784620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:36:58.784620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:58.784620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:58.784620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:36:58.784620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:36:58.784620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:36:58.784620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:36:58.784620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:36:58.784620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:36:58.784620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:36:58.784620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:36:58.784620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:36:58.784620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:36:58.784620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:36:58.784620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:36:58.784908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:36:58.784908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:36:58.784908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:36:58.784908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:36:58.784908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:36:58.784908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:00.833475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:00.834276 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:00.834378 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:00.835274 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:00.835477 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:00.835607 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:00.835737 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:00.835863 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:00.835970 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:00.836075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.836582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:00.836582 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:00.836582 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:00.836582 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:00.836582 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:00.836582 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:00.836582 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.836582 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:00.837302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.837302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:00.837302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:00.837302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:00.837302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.837302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:00.837302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:00.837302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.837302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:00.837302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:00.837302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:00.837302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.837509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:00.837509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:00.837509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:00.837509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:00.837509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:00.837509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:00.837509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:00.837509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:00.837509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:00.837509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:00.837509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:00.837509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:00.837509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:00.837724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.837724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:00.838010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.838010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:00.838010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:00.838010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:00.838010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:00.838010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:00.838010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.838010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:00.838010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:00.838010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.838010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:00.838010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:00.838010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.838010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:00.838010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:00.838010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.838289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:00.838289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:00.838289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.838289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:00.838289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:00.838289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:00.838289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.838289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:00.838289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:00.838289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:00.838289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:00.838289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:00.838289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:00.838289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:00.838289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:00.838289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:00.838289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:00.838289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:00.838289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:00.838579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:00.838579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.838579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:00.842989 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:00.843139 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:00.843268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:00.843495 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:37:00.843855 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:00.844022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.844474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:00.844474 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:00.844474 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:00.844474 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:00.844474 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:00.844474 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:00.844474 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.844474 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:00.844991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.844991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:00.844991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:00.844991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:00.844991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.844991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:00.844991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:00.844991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:00.844991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.844991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:00.844991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:00.844991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:00.844991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:00.844991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:00.844991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:00.845443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:00.845443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:00.845443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:00.845443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:00.845443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:00.845443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:00.845443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:00.845443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.845443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:00.845714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.845714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:00.845714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:00.845714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:00.845714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.845714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:00.845714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:00.845714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:37:00.845714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:00.845714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:00.845714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.845714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:00.846042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:00.846042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:00.846042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.846042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:00.846042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:00.846042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:00.846042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:00.846042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:00.846042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:00.846042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:00.846042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:00.846042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:00.846042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:00.846042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:00.846451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:00.846451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:00.846451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.846451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:00.846451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.846451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:00.846451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:00.846451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:00.846451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:00.846451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:00.846451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.846451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:00.846451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:00.846451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.846828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:00.846828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:00.846828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.846828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:00.846828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:00.846828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.846828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:00.846828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:00.846828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:37:00.846828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:00.846828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:00.846828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.846828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:00.846828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:00.846828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:00.846828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.846828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:00.846828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:00.846828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:00.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:00.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:00.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:00.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:00.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:00.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:00.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:00.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:00.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:00.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:00.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.847356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:00.849407 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:00.849643 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:00.849764 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:00.849885 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:00.850168 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:00.850408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.850878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:00.850878 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:00.850878 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:00.850878 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:00.850878 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:00.850878 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:00.850878 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.850878 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:00.851506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.851506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:00.851506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:00.851506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:00.851506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.851506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:00.851506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:00.851506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:00.851506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.851506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:00.851506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:00.851506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:00.851506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:00.851506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:00.851506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:00.851910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:00.851910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:00.851910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:00.851910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:00.851910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:00.851910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:00.851910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:00.851910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.851910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:00.852408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.852408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:00.852408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:00.852408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:00.852408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.852408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:00.852408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:00.852408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:00.852408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:00.852408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.852408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:00.852748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:00.852748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:00.852748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.852748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:00.852748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:00.852748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:00.852748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:00.852748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:00.852748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:00.852748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:00.852748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:00.852748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:00.852748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:00.852748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:00.853133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:00.853133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:00.853133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.853133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:00.853537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.853537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:00.853537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:00.853537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:00.853537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:00.853537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:00.853537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.853537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:00.853537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:00.853537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.853537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:00.853537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:00.853537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.853537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:00.853537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:00.853537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.853537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:00.854021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:00.854021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:00.854021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:00.854021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.854021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:00.854021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:00.854021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:00.854021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.854021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:00.854021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:00.854021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:00.854021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:00.854021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:00.854021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:00.854021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:00.854021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:00.854021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:00.854021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:00.854021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:00.854535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:00.854535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:00.854535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.854535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:00.875151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:00.875465 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:00.875555 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:00.875700 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:00.876024 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:00.876264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:00.876467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.876898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:00.876898 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:00.876898 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:00.876898 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:00.876898 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:00.876898 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:00.876898 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.876898 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:00.877544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.877544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:00.877544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:00.877544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:00.877544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.877544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:00.877544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:00.877544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.877544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:00.877544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:00.877544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:00.877544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.877902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:00.877902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:00.877902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:00.877902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:00.877902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:00.877902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:00.877902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:00.877902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:00.877902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:00.877902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:00.877902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:00.877902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:00.877902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:00.878278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.878278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:00.878278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.878278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:00.878278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:00.878278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:00.878278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:00.878278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:00.878278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.878278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:00.878278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:00.878278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.878278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:00.878278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:00.878278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.878697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:00.878697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:00.878697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:00.878697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:00.878697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:00.878697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.878697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:00.878697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:00.878697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:00.878697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:00.878697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:00.878697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:00.878697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:00.878697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:00.878697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:00.878697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:00.878697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:00.878697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:00.878697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:00.879207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:00.879207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:00.879207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:00.879207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:00.879207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:00.879207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:03.752207 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:03.752767 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:03.752767 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:03.753219 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:03.786036 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:03.786036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:03.786245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:03.786245 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:03.786245 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:03.786245 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:03.786245 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:03.786245 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:03.786245 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:03.786245 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:03.786963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:03.786963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:03.786963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:03.786963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:03.786963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:03.786963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:03.786963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:03.786963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:03.786963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:03.786963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:03.786963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:03.786963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:03.787427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:03.787427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:03.787427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:03.787427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:03.787427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:03.787427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:03.787427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:03.787427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:03.787427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:03.787427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:03.787427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:03.787427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:03.787427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:03.787934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:03.787934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:03.788093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:03.788093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:03.788093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:03.788093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:03.788093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:03.788093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:03.788093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:03.788093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:03.788093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:03.788093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:03.788093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:03.788093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:03.788093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:03.788093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:03.788093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:03.788093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:03.788544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:03.788544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:03.788544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:03.788544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:03.788544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:03.788544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:03.788544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:03.788544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:03.788544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:03.788544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:03.788544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:03.788544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:03.788544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:03.788544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:03.788544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:03.788544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:03.788544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:03.788544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:03.788544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:03.788909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:03.788909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:03.788909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:05.835279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:05.835279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:05.835279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:05.835803 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:05.835803 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:05.835953 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:05.836045 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:05.836192 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:05.836332 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:05.836461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.836948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:05.836948 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:05.836948 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:05.836948 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:05.836948 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:05.836948 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:05.836948 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.836948 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:05.840371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.840371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:05.840371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:05.840371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:05.840371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.840371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:05.840371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:05.840371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.840371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:05.840371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:05.840371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:05.840371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.840641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:05.840641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:05.840641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:05.840641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:05.840641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:05.840641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:05.840641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:05.840641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:05.840641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:05.840641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:05.840641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:05.840641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:05.840641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:05.840866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.840866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:05.840866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.840866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:05.840866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:05.840866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:05.840866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:05.840866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:05.840866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.840866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:05.840866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:05.840866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.840866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:05.840866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:05.840866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.841114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:05.841114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:05.841114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.841114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:05.841114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:05.841114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.841114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:05.841114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:05.841114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:05.841114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.841114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:05.841114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:05.841114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:05.841114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:05.841114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:05.841114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:05.841114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:05.841114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:05.841114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:05.841481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:05.841481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:05.841481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:05.841481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:05.841481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.841481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:05.843216 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:05.843445 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:05.843531 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:05.843648 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:37:05.843930 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:05.844118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.844537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:05.844537 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:05.844537 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:05.844537 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:05.844537 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:05.844537 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:05.844537 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.844537 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:05.845064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.845064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:05.845064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:05.845064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:05.845064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.845064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:05.845064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:05.845064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:05.845064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.845064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:05.845064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:05.845064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:05.845064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:05.845064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:05.845064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:05.845419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:05.845419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:05.845419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:05.845419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:05.845419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:05.845419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:05.845419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:05.845419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.845419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:05.845671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.845671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:05.845671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:05.845671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:05.845671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.845671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:05.845671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:05.845671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:37:05.845671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:05.845671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:05.845671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.845671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:05.846172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:05.846172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:05.846172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.846172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:05.846172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:05.846172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:05.846172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:05.846172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:05.846172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:05.846172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:05.846172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:05.846172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:05.846172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:05.846172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:05.846745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:05.846745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:05.846745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.846745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:05.846745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.846745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:05.846745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:05.846745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:05.846745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:05.846745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:05.846745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.846745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:05.846745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:05.846745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.847274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:05.847274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:05.847274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.847274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:05.847274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:05.847274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.847274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:05.847274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:05.847274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:37:05.847274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:05.847274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:05.847274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.847274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:05.847274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:05.847274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:05.847274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.847274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:05.847274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:05.847274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:05.847593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:05.847593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:05.847593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:05.847593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:05.847593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:05.847593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:05.847593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:05.847593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:05.847593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:05.847593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:05.847593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.847593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:05.849665 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:05.849857 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:05.849976 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:05.850064 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:05.850337 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:05.850546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.850954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:05.850954 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:05.850954 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:05.850954 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:05.850954 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:05.850954 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:05.850954 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.850954 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:05.851475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.851475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:05.851475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:05.851475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:05.851475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.851475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:05.851475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:05.851475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:05.851475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.851475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:05.851475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:05.851475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:05.851475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:05.851475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:05.851475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:05.851742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:05.851742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:05.851742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:05.851742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:05.851742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:05.851742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:05.851742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:05.851742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.851742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:05.852132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.852132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:05.852132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:05.852132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:05.852132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.852132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:05.852132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:05.852132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:05.852132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:05.852132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.852132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:05.852359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:05.852359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:05.852359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.852359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:05.852359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:05.852359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:05.852359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:05.852359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:05.852359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:05.852359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:05.852359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:05.852359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:05.852359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:05.852359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:05.852569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:05.852569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:05.852569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.852569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:05.852979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.852979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:05.852979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:05.852979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:05.852979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:05.852979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:05.852979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.852979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:05.852979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:05.852979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.852979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:05.852979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:05.852979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.852979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:05.852979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:05.852979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.852979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:05.853565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:05.853565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:05.853565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:05.853565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.853565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:05.853565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:05.853565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:05.853565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.853565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:05.853565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:05.853565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:05.853565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:05.853565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:05.853565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:05.853565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:05.853565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:05.853565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:05.853565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:05.853565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:05.853897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:05.853897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:05.853897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.853897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:05.875514 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:05.875782 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:05.875882 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:05.876034 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:05.876351 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:05.876558 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:05.876769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.877227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:05.877227 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:05.877227 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:05.877227 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:05.877227 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:05.877227 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:05.877227 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.877227 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:05.878079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.878079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:05.878079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:05.878079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:05.878079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.878079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:05.878079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:05.878079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.878079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:05.878079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:05.878079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:05.878079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.878462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:05.878462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:05.878462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:05.878462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:05.878462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:05.878462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:05.878462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:05.878462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:05.878462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:05.878462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:05.878462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:05.878462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:05.878462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:05.878873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.878873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:05.878873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.878873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:05.878873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:05.878873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:05.878873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:05.878873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:05.878873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.878873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:05.878873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:05.878873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.878873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:05.878873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:05.878873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.879331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:05.879331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:05.879331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:05.879331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:05.879331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:05.879331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.879331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:05.879331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:05.879331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:05.879331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:05.879331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:05.879331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:05.879331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:05.879331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:05.879331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:05.879331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:05.879331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:05.879331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:05.879331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:05.879892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:05.879892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:05.879892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:05.879892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:05.879892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:05.879892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:08.752664 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:08.753106 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:08.753106 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:08.753740 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:08.780093 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:08.780394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:08.780989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:08.780989 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:08.780989 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:08.780989 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:08.780989 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:08.780989 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:08.780989 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:08.780989 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:08.781711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:08.781711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:08.781711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:08.781711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:08.781711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:08.781711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:08.781711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:08.781711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:08.781711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:08.781711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:08.781711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:08.781711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:08.781980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:08.781980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:08.781980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:08.781980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:08.781980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:08.781980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:08.781980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:08.781980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:08.781980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:08.781980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:08.781980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:08.781980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:08.781980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:08.782203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:08.782203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:08.782381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:08.782381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:08.782381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:08.782381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:08.782381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:08.782381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:08.782381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:08.782381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:08.782381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:08.782381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:08.782381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:08.782381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:08.782381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:08.782381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:08.782381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:08.782381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:08.782740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:08.782740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:08.782740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:08.782740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:08.782740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:08.782740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:08.782740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:08.782740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:08.782740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:08.782740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:08.782740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:08.782740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:08.782740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:08.782740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:08.782740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:08.782740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:08.782740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:08.782740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:08.782740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:08.783091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:08.783091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:08.783091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:10.834919 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:10.835289 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:10.835405 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:10.836233 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:10.836490 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:10.836661 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:10.836813 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:10.836965 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:10.837110 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:10.837259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.837843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:10.837843 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:10.837843 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:10.837843 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:10.837843 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:10.837843 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:10.837843 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.837843 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:10.839123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.839123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:10.839123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:10.839123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:10.839123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.839123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:10.839123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:10.839123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.839123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:10.839123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:10.839123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:10.839123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.839389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:10.839389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:10.839389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:10.839389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:10.839389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:10.839389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:10.839389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:10.839389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:10.839389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:10.839389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:10.839389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:10.839389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:10.839389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:10.839673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.839673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:10.839673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.839673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:10.839673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:10.839673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:10.839673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:10.839673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:10.839673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.839673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:10.839673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:10.839673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.839673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:10.839673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:10.839673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.839966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:10.839966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:10.839966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.839966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:10.839966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:10.839966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.839966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:10.839966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:10.839966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:10.839966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.839966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:10.839966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:10.839966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:10.839966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:10.839966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:10.839966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:10.839966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:10.839966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:10.839966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:10.840283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:10.840283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:10.840283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:10.840283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:10.840283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.840283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:10.844527 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:10.844779 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:10.844867 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:10.844984 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:37:10.845428 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:10.845655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.846163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:10.846163 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:10.846163 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:10.846163 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:10.846163 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:10.846163 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:10.846163 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.846163 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:10.846716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.846716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:10.846716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:10.846716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:10.846716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.846716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:10.846716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:10.846716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:10.846716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.846716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:10.846716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:10.846716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:10.846716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:10.846716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:10.846716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:10.847021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:10.847021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:10.847021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:10.847021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:10.847021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:10.847021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:10.847021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:10.847021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.847021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:10.847285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.847285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:10.847285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:10.847285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:10.847285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.847285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:10.847285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:10.847285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:37:10.847285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:10.847285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:10.847285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.847285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:10.847516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:10.847516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:10.847516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.847516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:10.847516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:10.847516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:10.847516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:10.847516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:10.847516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:10.847516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:10.847516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:10.847516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:10.847516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:10.847516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:10.847778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:10.847778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:10.847778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.847778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:10.848047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.848047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:10.848047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:10.848047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:10.848047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:10.848047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:10.848047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.848047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:10.848047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:10.848047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.848047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:10.848047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:10.848047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.848047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:10.848047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:10.848047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.848337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:10.848337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:10.848337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:37:10.848337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:10.848337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:10.848337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.848337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:10.848337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:10.848337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:10.848337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.848337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:10.848337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:10.848337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:10.848337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:10.848337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:10.848337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:10.848337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:10.848337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:10.848629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:10.848629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:10.848629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:10.848629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:10.848629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:10.848629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.848629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:10.849970 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:10.850164 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:10.850265 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:10.850365 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:10.850748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:10.851000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.851531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:10.851531 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:10.851531 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:10.851531 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:10.851531 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:10.851531 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:10.851531 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.851531 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:10.852041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.852041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:10.852041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:10.852041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:10.852041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.852041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:10.852041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:10.852041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:10.852041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.852041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:10.852041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:10.852041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:10.852041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:10.852041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:10.852041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:10.852318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:10.852318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:10.852318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:10.852318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:10.852318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:10.852318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:10.852318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:10.852318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.852318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:10.852637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.852637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:10.852637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:10.852637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:10.852637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.852637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:10.852637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:10.852637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:10.852637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:10.852637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.852637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:10.852862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:10.852862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:10.852862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.852862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:10.852862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:10.852862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:10.852862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:10.852862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:10.852862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:10.852862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:10.852862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:10.852862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:10.852862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:10.852862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:10.853082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:10.853082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:10.853082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.853082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:10.853350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.853350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:10.853350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:10.853350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:10.853350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:10.853350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:10.853350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.853350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:10.853350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:10.853350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.853350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:10.853350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:10.853350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.853350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:10.853350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:10.853350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.853350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:10.853651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:10.853651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:10.853651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:10.853651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.853651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:10.853651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:10.853651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:10.853651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.853651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:10.853651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:10.853651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:10.853651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:10.853651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:10.853651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:10.853651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:10.853651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:10.853651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:10.853651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:10.853651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:10.853986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:10.853986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:10.853986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.853986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:10.875935 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:10.876217 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:10.876326 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:10.876454 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:10.876901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:10.877173 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:10.877437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.877990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:10.877990 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:10.877990 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:10.877990 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:10.877990 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:10.877990 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:10.877990 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.877990 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:10.878736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.878736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:10.878736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:10.878736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:10.878736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.878736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:10.878736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:10.878736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.878736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:10.878736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:10.878736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:10.878736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.879217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:10.879217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:10.879217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:10.879217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:10.879217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:10.879217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:10.879217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:10.879217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:10.879217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:10.879217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:10.879217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:10.879217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:10.879217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:10.879479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.879479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:10.879479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.879479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:10.879479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:10.879479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:10.879479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:10.879479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:10.879479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.879479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:10.879479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:10.879479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.879479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:10.879479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:10.879479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.879759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:10.879759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:10.879759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:10.879759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:10.879759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:10.879759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.879759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:10.879759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:10.879759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:10.879759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:10.879759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:10.879759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:10.879759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:10.879759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:10.879759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:10.879759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:10.879759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:10.879759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:10.879759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:10.880064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:10.880064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:10.880064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:10.880064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:10.880064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:10.880064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:13.753031 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:13.753675 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:13.753675 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:13.754182 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:13.783399 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:13.783634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:13.784095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:13.784095 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:13.784095 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:13.784095 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:13.784095 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:13.784095 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:13.784095 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:13.784095 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:13.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:13.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:13.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:13.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:13.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:13.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:13.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:13.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:13.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:13.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:13.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:13.784716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:13.785590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:13.785590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:13.785590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:13.785590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:13.785590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:13.785590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:13.785590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:13.785590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:13.785590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:13.785590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:13.785590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:13.785590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:13.785590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:13.785902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:13.785902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:13.785902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:13.785902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:13.785902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:13.785902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:13.785902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:13.785902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:13.785902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:13.785902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:13.785902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:13.785902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:13.785902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:13.785902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:13.785902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:13.786234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:13.786234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:13.786234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:13.786234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:13.786234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:13.786234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:13.786234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:13.786234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:13.786234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:13.786234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:13.786234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:13.786234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:13.786234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:13.786234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:13.786234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:13.786234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:13.786234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:13.786234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:13.786234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:13.786674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:13.786674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:13.786674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:13.786674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:13.786674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:13.786674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:15.835527 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:15.835970 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:15.835970 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:15.836769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:15.836937 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:15.837064 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:15.837193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:15.837340 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:15.843685 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:15.843685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.843685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:15.843685 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:15.843685 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:15.843685 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:15.843685 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:15.843685 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:15.843685 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.843685 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:15.843685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.843685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:15.843685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:15.843685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:15.843979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.843979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:15.843979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:15.843979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.843979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:15.843979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:15.843979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:15.843979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.843979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:15.843979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:15.843979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:15.843979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:15.843979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:15.843979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:15.844225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:15.844225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:15.844225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:15.844225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:15.844225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:15.844225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:15.844225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:15.844225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.844225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:15.844225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.844225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:15.844225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:15.844473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.844834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:15.844834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:15.844834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:15.844834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:15.844834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:15.844834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:15.844834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:15.844834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:15.844834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:15.844834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:15.844834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:15.844834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:15.844834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:15.844834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.844834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:15.845062 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:15.845393 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:15.845437 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:15.845544 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:37:15.845980 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:15.846171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.846748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:15.846748 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:15.846748 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:15.846748 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:15.846748 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:15.846748 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:15.846748 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.846748 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:15.847478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.847478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:15.847478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:15.847478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:15.847478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.847478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:15.847478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:15.847478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:15.847478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.847478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:15.847478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:15.847478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:15.847478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:15.847478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:15.847478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:15.847888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:15.847888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:15.847888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:15.847888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:15.847888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:15.847888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:15.847888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:15.847888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.847888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:15.848341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.848341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:15.848341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:15.848341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:15.848341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.848341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:15.848341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:15.848341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:37:15.848341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:15.848341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:15.848341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.848341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:15.848678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:15.848678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:15.848678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.848678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:15.848678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:15.848678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:15.848678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:15.848678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:15.848678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:15.848678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:15.848678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:15.848678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:15.848678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:15.848678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:15.849021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:15.849021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:15.849021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.849021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:15.849374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.849374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:15.849374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:15.849374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:15.849374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:15.849374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:15.849374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.849374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:15.849374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:15.849374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.849374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:15.849374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:15.849374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.849374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:15.849374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:15.849374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.849797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:15.849797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:15.849797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:37:15.849797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:15.849797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:15.849797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.849797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:15.849797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:15.849797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:15.849797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.849797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:15.849797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:15.849797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:15.849797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:15.849797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:15.849797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:15.849797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:15.849797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:15.850232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:15.850232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:15.850232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:15.850232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:15.850232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:15.850232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.850232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:15.850417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:15.850665 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:15.850774 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:15.850900 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:15.851235 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:15.851595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.851913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:15.851913 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:15.851913 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:15.851913 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:15.851913 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:15.851913 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:15.851913 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.851913 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:15.852626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.852626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:15.852626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:15.852626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:15.852626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.852626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:15.852626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:15.852626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:15.852626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.852626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:15.852626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:15.852626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:15.852626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:15.852626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:15.852626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:15.852902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:15.852902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:15.852902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:15.852902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:15.852902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:15.852902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:15.852902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:15.852902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.852902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:15.853481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.853481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:15.853481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:15.853481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:15.853481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.853481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:15.853481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:15.853481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:15.853481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:15.853481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.853481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:15.853706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:15.853706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:15.853706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.853706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:15.853706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:15.853706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:15.853706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:15.853706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:15.853706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:15.853706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:15.853706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:15.853706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:15.853706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:15.853706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:15.853930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:15.853930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:15.853930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.853930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:15.854782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.854782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:15.854782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:15.854782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:15.854782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:15.854782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:15.854782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.854782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:15.854782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:15.854782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.854782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:15.854782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:15.854782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.854782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:15.854782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:15.854782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.854782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:15.855151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:15.855151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:15.855151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:15.855151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.855151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:15.855151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:15.855151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:15.855151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.855151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:15.855151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:15.855151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:15.855151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:15.855151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:15.855151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:15.855151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:15.855151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:15.855151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:15.855151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:15.855151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:15.855547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:15.855547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:15.855547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.855547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:15.876420 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:15.876850 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:15.876967 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:15.877148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:15.877479 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:15.877718 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:15.877939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.878541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:15.878541 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:15.878541 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:15.878541 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:15.878541 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:15.878541 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:15.878541 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.878541 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:15.879361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.879361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:15.879361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:15.879361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:15.879361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.879361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:15.879361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:15.879361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.879361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:15.879361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:15.879361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:15.879361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.879612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:15.879612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:15.879612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:15.879612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:15.879612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:15.879612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:15.879612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:15.879612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:15.879612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:15.879612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:15.879612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:15.879612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:15.879612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:15.879819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.879819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:15.880566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.880566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:15.880566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:15.880566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:15.880566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:15.880566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:15.880566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.880566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:15.880566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:15.880566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.880566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:15.880566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:15.880566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.880566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:15.880566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:15.880566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:15.880842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:15.880842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:15.880842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.880842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:15.880842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:15.880842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:15.880842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:15.880842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:15.880842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:15.880842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:15.880842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:15.880842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:15.880842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:15.880842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:15.880842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:15.880842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:15.880842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:15.880842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:15.880842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:15.881156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:15.881156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:15.881156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:18.753479 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:18.753956 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:18.753956 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:18.754501 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:18.790010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:18.790190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:18.790658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:18.790658 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:18.790658 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:18.790658 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:18.790658 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:18.790658 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:18.790658 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:18.790658 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:18.791286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:18.791286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:18.791286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:18.791286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:18.791286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:18.791286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:18.791286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:18.791286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:18.791286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:18.791286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:18.791286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:18.791286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:18.791893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:18.791893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:18.791893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:18.791893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:18.791893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:18.791893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:18.791893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:18.791893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:18.791893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:18.791893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:18.791893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:18.791893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:18.791893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:18.792154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:18.792154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:18.792154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:18.792154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:18.792154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:18.792154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:18.792154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:18.792154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:18.792154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:18.792154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:18.792154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:18.792154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:18.792154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:18.792154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:18.792154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:18.792422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:18.792422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:18.792422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:18.792422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:18.792422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:18.792422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:18.792422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:18.792422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:18.792422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:18.792422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:18.792422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:18.792422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:18.792422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:18.792422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:18.792422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:18.792422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:18.792422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:18.792422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:18.792422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:18.792769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:18.792769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:18.792769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:18.792769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:18.792769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:18.792769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:20.835865 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:20.836538 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:20.836664 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:20.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:20.840403 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:20.840521 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:20.840650 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:20.840757 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:20.840956 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:20.841053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.841554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:20.841554 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:20.841554 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:20.841554 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:20.841554 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:20.841554 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:20.841554 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.841554 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:20.842141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.842141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:20.842141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:20.842141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:20.842141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.842141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:20.842141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:20.842141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.842141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:20.842141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:20.842141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:20.842141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.842386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:20.842386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:20.842386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:20.842386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:20.842386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:20.842386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:20.842386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:20.842386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:20.842386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:20.842386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:20.842386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:20.842386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:20.842386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:20.842602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.842602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:20.842847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.842847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:20.842847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:20.842847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:20.842847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:20.842847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:20.842847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.842847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:20.842847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:20.842847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.842847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:20.842847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:20.842847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.842847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:20.842847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:20.842847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.843127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:20.843127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:20.843127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.843127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:20.843127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:20.843127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:20.843127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.843127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:20.843127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:20.843127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:20.843127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:20.843127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:20.843127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:20.843127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:20.843127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:20.843127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:20.843127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:20.843127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:20.843127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:20.843429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:20.843429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.843429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:20.845244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:20.847524 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:20.847616 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:20.847710 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:37:20.850688 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:20.851536 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:20.851732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.852201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:20.852201 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:20.852201 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:20.852201 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:20.852201 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:20.852201 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:20.852201 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.852201 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:20.852736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.852736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:20.852736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:20.852736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:20.852736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.852736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:20.852736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:20.852736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:20.852736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.852736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:20.852736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:20.852736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:20.852736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:20.852736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:20.852736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:20.853148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:20.853148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:20.853148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:20.853148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:20.853148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:20.853148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:20.853148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:20.853148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.853148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:20.853431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.853431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:20.853431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:20.853431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:20.853431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.853431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:20.853431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:20.853431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:37:20.853431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:20.853431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:20.853431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.853431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:20.853805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:20.853805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:20.853805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.853805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:20.853805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:20.853805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:20.853805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:20.853805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:20.853805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:20.853805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:20.853805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:20.853805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:20.853805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:20.853805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:20.854324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:20.854324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:20.854324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.854324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:20.854324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.854324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:20.854324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:20.854324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:20.854324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:20.854324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:20.854324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.854324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:20.854324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:20.854324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.854742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:20.854742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:20.854742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.854742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:20.854742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:20.854742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.854742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:20.854742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:20.854742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:37:20.854742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:20.854742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:20.854742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.854742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:20.854742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:20.854742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:20.854742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.854742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:20.854742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:20.854742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:20.855273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:20.855273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:20.855273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:20.855273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:20.855273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:20.855273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:20.855273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:20.855273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:20.855273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:20.855273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:20.855273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.855273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:20.855273 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:20.855273 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:20.855680 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:20.855680 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:20.855680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.858419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:20.858419 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:20.858419 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:20.858419 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:20.858419 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:20.858419 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:20.858419 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.858419 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:20.858936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.858936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:20.858936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:20.858936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:20.858936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.858936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:20.858936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:20.858936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:20.858936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.858936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:20.858936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:20.858936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:20.858936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:20.858936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:20.858936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:20.859377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:20.859377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:20.859377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:20.859377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:20.859377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:20.859377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:20.859377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:20.859377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.859377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:20.859631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.859631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:20.859631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:20.859631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:20.859631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.859631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:20.859631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:20.859631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:20.859631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:20.859631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.859631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:20.860238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:20.860238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:20.860238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.860238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:20.860238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:20.860238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:20.860238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:20.860238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:20.860238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:20.860238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:20.860238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:20.860238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:20.860238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:20.860238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:20.860696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:20.860696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:20.860696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.860696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:20.860696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.860696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:20.860696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:20.860696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:20.860696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:20.860696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:20.860696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.860696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:20.860696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:20.860696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.861082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:20.861082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:20.861082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.861082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:20.861082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:20.861082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.861082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:20.861082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:20.861082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:20.861082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:20.861082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.861082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:20.861082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:20.861082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:20.861082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.861082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:20.861082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:20.861082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:20.861082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:20.861652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:20.861652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:20.861652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:20.861652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:20.861652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:20.861652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:20.861652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:20.861652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:20.861652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:20.861652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.861652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:20.876823 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:20.877057 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:20.877149 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:20.877299 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:20.877555 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:20.877737 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:20.877901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.878239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:20.878239 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:20.878239 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:20.878239 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:20.878239 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:20.878239 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:20.878239 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.878239 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:20.878807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.878807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:20.878807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:20.878807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:20.878807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.878807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:20.878807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:20.878807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.878807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:20.878807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:20.878807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:20.878807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.879044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:20.879044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:20.879044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:20.879044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:20.879044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:20.879044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:20.879044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:20.879044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:20.879044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:20.879044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:20.879044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:20.879044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:20.879044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:20.879292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.879292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:20.879582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.879582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:20.879582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:20.879582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:20.879582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:20.879582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:20.879582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.879582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:20.879582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:20.879582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.879582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:20.879582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:20.879582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.879582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:20.879582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:20.879582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:20.879864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:20.879864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:20.879864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.879864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:20.879864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:20.879864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:20.879864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:20.879864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:20.879864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:20.879864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:20.879864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:20.879864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:20.879864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:20.879864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:20.879864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:20.879864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:20.879864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:20.879864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:20.879864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:20.880282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:20.880282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:20.880282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:23.753883 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:23.754449 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:23.754449 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:23.755098 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:23.788201 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:23.788433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:23.788917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:23.788917 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:23.788917 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:23.788917 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:23.788917 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:23.788917 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:23.788917 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:23.788917 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:23.789662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:23.789662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:23.789662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:23.789662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:23.789662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:23.789662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:23.789662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:23.789662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:23.789662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:23.789662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:23.789662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:23.789662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:23.790198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:23.790198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:23.790198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:23.790198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:23.790198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:23.790198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:23.790198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:23.790198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:23.790198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:23.790198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:23.790198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:23.790198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:23.790198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:23.790739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:23.790739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:23.790832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:23.790832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:23.790832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:23.790832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:23.790832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:23.790832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:23.790832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:23.790832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:23.790832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:23.790832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:23.790832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:23.790832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:23.790832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:23.790832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:23.790832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:23.790832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:23.791175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:23.791175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:23.791175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:23.791175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:23.791175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:23.791175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:23.791175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:23.791175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:23.791175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:23.791175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:23.791175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:23.791175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:23.791175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:23.791175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:23.791175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:23.791175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:23.791175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:23.791175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:23.791175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:23.791566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:23.791566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:23.791566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:25.836680 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:25.837158 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:25.837158 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:25.837920 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:25.838064 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:25.838186 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:25.838294 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:25.838407 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:25.838513 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:25.838656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.839314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:25.839314 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:25.839314 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:25.839314 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:25.839314 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:25.839314 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:25.839314 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.839314 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:25.840063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.840063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:25.840063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:25.840063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:25.840063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.840063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:25.840063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:25.840063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.840063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:25.840063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:25.840063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:25.840063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.840434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:25.840434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:25.840434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:25.840434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:25.840434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:25.840434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:25.840434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:25.840434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:25.840434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:25.840434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:25.840434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:25.840434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:25.840434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:25.840689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.840689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:25.840872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.840872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:25.840872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:25.840872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:25.840872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:25.840872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:25.840872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.840872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:25.840872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:25.840872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.840872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:25.840872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:25.840872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.840872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:25.840872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:25.840872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.841158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:25.841158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:25.841158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.841158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:25.841158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:25.841158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:25.841158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.841158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:25.841158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:25.841158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:25.841158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:25.841158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:25.841158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:25.841158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:25.841158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:25.841158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:25.841158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:25.841158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:25.841158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:25.841496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:25.841496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.841496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:25.847697 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:25.847955 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:25.848062 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:25.848157 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:37:25.848477 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:25.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.848957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:25.848957 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:25.848957 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:25.848957 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:25.848957 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:25.848957 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:25.848957 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.848957 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:25.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:25.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:25.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:25.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:25.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:25.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:25.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:25.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:25.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:25.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:25.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:25.849492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:25.849773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:25.849773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:25.849773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:25.849773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:25.849773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:25.849773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:25.849773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:25.849773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.849773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:25.850083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.850083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:25.850083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:25.850083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:25.850083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.850083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:25.850083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:25.850083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:37:25.850083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:25.850083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:25.850083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.850083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:25.850340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:25.850340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:25.850340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.850340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:25.850340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:25.850340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:25.850340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:25.850340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:25.850340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:25.850340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:25.850340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:25.850340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:25.850340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:25.850340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:25.850568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:25.850568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:25.850568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.850568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:25.850802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.850802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:25.850802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:25.850802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:25.850802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:25.850802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:25.850802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.850802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:25.850802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:25.850802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.850802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:25.850802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:25.850802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.850802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:25.850802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:25.850802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.851105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:25.851105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:25.851105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:37:25.851105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:25.851105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:25.851105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.851105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:25.851105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:25.851105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:25.851105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.851105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:25.851105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:25.851105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:25.851105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:25.851105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:25.851105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:25.851105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:25.851105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:25.851400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:25.851400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:25.851400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:25.851400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:25.851400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:25.851400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.851400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:25.854240 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:25.854498 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:25.854603 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:25.854699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:25.854931 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:25.855116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.855471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:25.855471 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:25.855471 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:25.855471 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:25.855471 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:25.855471 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:25.855471 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.855471 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:25.855956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.855956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:25.855956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:25.855956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:25.855956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.855956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:25.855956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:25.855956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:25.855956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.855956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:25.855956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:25.855956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:25.855956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:25.855956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:25.855956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:25.856207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:25.856207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:25.856207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:25.856207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:25.856207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:25.856207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:25.856207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:25.856207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.856207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:25.856545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.856545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:25.856545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:25.856545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:25.856545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.856545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:25.856545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:25.856545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:25.856545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:25.856545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.856545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:25.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:25.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:25.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:25.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:25.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:25.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:25.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:25.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:25.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:25.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:25.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:25.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:25.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:25.857092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:25.857092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:25.857092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.857092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:25.857261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.857261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:25.857261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:25.857261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:25.857261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:25.857261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:25.857261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.857261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:25.857261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:25.857261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.857261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:25.857261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:25.857261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.857261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:25.857261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:25.857261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.857261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:25.857552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:25.857552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:25.857552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:25.857552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.857552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:25.857552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:25.857552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:25.857552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.857552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:25.857552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:25.857552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:25.857552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:25.857552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:25.857552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:25.857552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:25.857552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:25.857552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:25.857552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:25.857552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:25.857853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:25.857853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:25.857853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.857853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:25.877275 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:25.877535 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:25.877642 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:25.877833 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:25.878073 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:25.878264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:25.878456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.878878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:25.878878 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:25.878878 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:25.878878 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:25.878878 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:25.878878 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:25.878878 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.878878 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:25.879487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.879487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:25.879487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:25.879487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:25.879487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.879487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:25.879487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:25.879487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.879487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:25.879487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:25.879487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:25.879487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.879788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:25.879788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:25.879788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:25.879788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:25.879788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:25.879788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:25.879788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:25.879788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:25.879788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:25.879788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:25.879788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:25.879788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:25.879788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:25.880019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.880019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:25.880138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.880138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:25.880138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:25.880138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:25.880138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:25.880138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:25.880138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.880138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:25.880138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:25.880138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.880138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:25.880138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:25.880138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.880138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:25.880138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:25.880138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:25.880464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:25.880464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:25.880464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.880464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:25.880464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:25.880464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:25.880464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:25.880464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:25.880464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:25.880464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:25.880464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:25.880464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:25.880464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:25.880464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:25.880464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:25.880464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:25.880464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:25.880464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:25.880464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:25.880757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:25.880757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:25.880757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:28.754285 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:28.754753 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:28.754753 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:28.755550 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:28.794830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:28.795071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:28.795657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:28.795657 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:28.795657 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:28.795657 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:28.795657 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:28.795657 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:28.795657 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:28.795657 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:28.796634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:28.796634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:28.796634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:28.796634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:28.796634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:28.796634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:28.796634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:28.796634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:28.796634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:28.796634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:28.796634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:28.796634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:28.796989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:28.796989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:28.796989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:28.796989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:28.796989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:28.796989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:28.796989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:28.796989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:28.796989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:28.796989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:28.796989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:28.796989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:28.796989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:28.797244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:28.797244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:28.797757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:28.797757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:28.797757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:28.797757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:28.797757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:28.797757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:28.797757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:28.797757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:28.797757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:28.797757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:28.797757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:28.797757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:28.797757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:28.797757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:28.797757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:28.797757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:28.798090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:28.798090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:28.798090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:28.798090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:28.798090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:28.798090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:28.798090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:28.798090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:28.798090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:28.798090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:28.798090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:28.798090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:28.798090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:28.798090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:28.798090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:28.798090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:28.798090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:28.798090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:28.798090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:28.798422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:28.798422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:28.798422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:30.837072 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:30.837571 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:30.837571 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:30.838423 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:30.838561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:30.838708 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:30.838816 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:30.838904 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:30.839000 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:30.839094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.839513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:30.839513 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:30.839513 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:30.839513 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:30.839513 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:30.839513 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:30.839513 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.839513 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:30.840095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.840095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:30.840095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:30.840095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:30.840095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.840095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:30.840095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:30.840095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.840095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:30.840095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:30.840095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:30.840095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.840337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:30.840337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:30.840337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:30.840337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:30.840337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:30.840337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:30.840337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:30.840337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:30.840337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:30.840337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:30.840337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:30.840337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:30.840337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:30.840571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.840571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:30.840862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.840862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:30.840862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:30.840862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:30.840862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:30.840862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:30.840862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.840862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:30.840862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:30.840862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.840862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:30.840862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:30.840862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.840862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:30.840862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:30.840862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.841137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:30.841137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:30.841137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.841137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:30.841137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:30.841137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:30.841137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.841137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:30.841137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:30.841137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:30.841137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:30.841137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:30.841137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:30.841137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:30.841137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:30.841137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:30.841137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:30.841137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:30.841137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:30.841507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:30.841507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.841507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:30.848032 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:30.848261 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:30.848364 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:30.848462 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:37:30.848736 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:30.848903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.849543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:30.849543 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:30.849543 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:30.849543 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:30.849543 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:30.849543 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:30.849543 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.849543 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:30.850146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.850146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:30.850146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:30.850146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:30.850146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.850146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:30.850146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:30.850146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:30.850146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.850146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:30.850146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:30.850146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:30.850146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:30.850146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:30.850146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:30.850427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:30.850427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:30.850427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:30.850427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:30.850427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:30.850427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:30.850427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:30.850427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.850427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:30.850763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.850763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:30.850763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:30.850763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:30.850763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.850763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:30.850763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:30.850763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:37:30.850763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:30.850763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:30.850763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.850763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:30.851014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:30.851014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:30.851014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.851014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:30.851014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:30.851014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:30.851014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:30.851014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:30.851014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:30.851014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:30.851014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:30.851014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:30.851014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:30.851014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:30.851289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:30.851289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:30.851289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.851289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:30.851435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.851435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:30.851435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:30.851435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:30.851435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:30.851435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:30.851435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.851435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:30.851435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:30.851435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.851435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:30.851435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:30.851435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.851435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:30.851435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:30.851435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.851736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:30.851736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:30.851736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:37:30.851736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:30.851736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:30.851736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.851736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:30.851736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:30.851736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:30.851736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.851736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:30.851736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:30.851736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:30.851736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:30.851736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:30.851736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:30.851736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:30.851736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:30.852032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:30.852032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:30.852032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:30.852032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:30.852032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:30.852032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.852032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:30.854598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:30.854853 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:30.854935 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:30.855298 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:30.855378 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:30.855665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.856222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:30.856222 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:30.856222 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:30.856222 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:30.856222 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:30.856222 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:30.856222 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.856222 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:30.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:30.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:30.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:30.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:30.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:30.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:30.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:30.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:30.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:30.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:30.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:30.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:30.857188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:30.857188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:30.857188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:30.857188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:30.857188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:30.857188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:30.857188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:30.857188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.857188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:30.857486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.857486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:30.857486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:30.857486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:30.857486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.857486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:30.857486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:30.857486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:30.857486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:30.857486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.857486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:30.857862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:30.857862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:30.857862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.857862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:30.857862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:30.857862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:30.857862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:30.857862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:30.857862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:30.857862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:30.857862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:30.857862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:30.857862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:30.857862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:30.858658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:30.858658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:30.858658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.858658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:30.858658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.858658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:30.858658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:30.858658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:30.858658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:30.858658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:30.858658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.858658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:30.858658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:30.858658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.859015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:30.859015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:30.859015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.859015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:30.859015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:30.859015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.859015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:30.859015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:30.859015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:30.859015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:30.859015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.859015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:30.859015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:30.859015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:30.859015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.859015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:30.859015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:30.859015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:30.859015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:30.859520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:30.859520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:30.859520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:30.859520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:30.859520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:30.859520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:30.859520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:30.859520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:30.859520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:30.859520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.859520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:30.877678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:30.877969 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:30.878050 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:30.878188 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:30.878497 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:30.878727 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:30.878896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.879268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:30.879268 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:30.879268 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:30.879268 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:30.879268 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:30.879268 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:30.879268 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.879268 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:30.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:30.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:30.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:30.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:30.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:30.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:30.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:30.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:30.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.880244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:30.880244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:30.880244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:30.880244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:30.880244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:30.880244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:30.880244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:30.880244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:30.880244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:30.880244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:30.880244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:30.880244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:30.880244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:30.880501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.880501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:30.880825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.880825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:30.880825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:30.880825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:30.880825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:30.880825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:30.880825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.880825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:30.880825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:30.880825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.880825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:30.880825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:30.880825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.880825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:30.880825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:30.880825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:30.881291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:30.881291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:30.881291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.881291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:30.881291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:30.881291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:30.881291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:30.881291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:30.881291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:30.881291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:30.881291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:30.881291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:30.881291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:30.881291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:30.881291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:30.881291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:30.881291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:30.881291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:30.881291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:30.881857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:30.881857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:30.881857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:33.754768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:33.755243 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:33.755243 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:33.755901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:33.796172 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:33.796759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:33.797536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:33.797536 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:33.797536 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:33.797536 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:33.797536 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:33.797536 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:33.797536 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:33.797536 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:33.798382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:33.798382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:33.798382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:33.798382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:33.798382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:33.798382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:33.798382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:33.798382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:33.798382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:33.798382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:33.798382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:33.798382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:33.798700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:33.798700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:33.798700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:33.798700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:33.798700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:33.798700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:33.798700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:33.798700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:33.798700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:33.798700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:33.798700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:33.798700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:33.798700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:33.798989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:33.798989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:33.799208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:33.799208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:33.799208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:33.799208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:33.799208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:33.799208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:33.799208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:33.799208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:33.799208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:33.799208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:33.799208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:33.799208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:33.799208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:33.799208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:33.799208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:33.799208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:33.799546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:33.799546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:33.799546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:33.799546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:33.799546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:33.799546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:33.799546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:33.799546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:33.799546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:33.799546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:33.799546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:33.799546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:33.799546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:33.799546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:33.799546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:33.799546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:33.799546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:33.799546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:33.799546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:33.799947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:33.799947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:33.799947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:35.837541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:35.838111 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:35.838111 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:35.838641 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:35.838798 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:35.839057 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:35.839125 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:35.839316 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:35.839432 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:35.839593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.840065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:35.840065 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:35.840065 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:35.840065 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:35.840065 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:35.840065 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:35.840065 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.840065 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:35.840698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.840698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:35.840698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:35.840698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:35.840698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.840698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:35.840698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:35.840698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.840698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:35.840698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:35.840698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:35.840698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.840929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:35.840929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:35.840929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:35.840929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:35.840929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:35.840929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:35.840929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:35.840929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:35.840929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:35.840929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:35.840929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:35.840929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:35.840929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:35.841129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.841129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:35.841402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.841402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:35.841402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:35.841402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:35.841402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:35.841402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:35.841402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.841402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:35.841402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:35.841402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.841402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:35.841402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:35.841402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.841402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:35.841402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:35.841402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.841671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:35.841671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:35.841671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.841671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:35.841671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:35.841671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:35.841671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.841671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:35.841671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:35.841671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:35.841671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:35.841671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:35.841671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:35.841671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:35.841671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:35.841671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:35.841671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:35.841671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:35.841671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:35.841948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:35.841948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.841948 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:35.848417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:35.848712 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:35.848800 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:35.848890 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:37:35.849223 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:35.849434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.849874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:35.849874 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:35.849874 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:35.849874 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:35.849874 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:35.849874 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:35.849874 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.849874 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:35.850454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.850454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:35.850454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:35.850454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:35.850454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.850454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:35.850454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:35.850454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:35.850454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.850454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:35.850454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:35.850454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:35.850454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:35.850454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:35.850454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:35.850742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:35.850742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:35.850742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:35.850742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:35.850742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:35.850742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:35.850742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:35.850742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.850742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:35.851011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.851011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:35.851011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:35.851011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:35.851011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.851011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:35.851011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:35.851011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:37:35.851011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:35.851011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:35.851011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.851011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:35.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:35.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:35.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:35.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:35.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:35.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:35.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:35.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:35.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:35.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:35.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:35.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:35.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:35.851492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:35.851492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:35.851492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.851492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:35.851755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.851755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:35.851755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:35.851755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:35.851755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:35.851755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:35.851755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.851755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:35.851755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:35.851755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.851755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:35.851755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:35.851755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.851755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:35.851755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:35.851755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.852014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:35.852014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:35.852014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:37:35.852014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:35.852014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:35.852014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.852014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:35.852014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:35.852014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:35.852014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.852014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:35.852014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:35.852014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:35.852014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:35.852014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:35.852014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:35.852014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:35.852014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:35.852303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:35.852303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:35.852303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:35.852303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:35.852303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:35.852303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.852303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:35.854878 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:35.855097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:35.855177 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:35.855308 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:35.855637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:35.855849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.856238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:35.856238 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:35.856238 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:35.856238 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:35.856238 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:35.856238 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:35.856238 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.856238 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:35.856760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.856760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:35.856760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:35.856760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:35.856760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.856760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:35.856760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:35.856760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:35.856760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.856760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:35.856760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:35.856760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:35.856760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:35.856760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:35.856760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:35.857004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:35.857004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:35.857004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:35.857004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:35.857004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:35.857004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:35.857004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:35.857004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.857004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:35.857370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.857370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:35.857370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:35.857370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:35.857370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.857370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:35.857370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:35.857370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:35.857370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:35.857370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.857370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:35.857699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:35.857699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:35.857699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.857699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:35.857699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:35.857699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:35.857699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:35.857699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:35.857699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:35.857699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:35.857699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:35.857699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:35.857699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:35.857699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:35.857904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:35.857904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:35.857904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.857904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:35.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:35.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:35.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:35.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:35.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:35.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:35.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:35.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:35.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:35.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:35.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:35.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:35.858408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:35.858408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:35.858408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:35.858408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.858408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:35.858408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:35.858408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:35.858408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.858408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:35.858408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:35.858408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:35.858408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:35.858408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:35.858408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:35.858408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:35.858408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:35.858408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:35.858408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:35.858408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:35.858797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:35.858797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:35.858797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.858797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:35.878062 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:35.878350 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:35.878477 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:35.878685 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:35.878968 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:35.879149 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:35.879353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.879791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:35.879791 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:35.879791 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:35.879791 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:35.879791 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:35.879791 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:35.879791 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.879791 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:35.880698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.880698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:35.880698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:35.880698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:35.880698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.880698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:35.880698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:35.880698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.880698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:35.880698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:35.880698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:35.880698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:35.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:35.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:35.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:35.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:35.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:35.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:35.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:35.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:35.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:35.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:35.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:35.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:35.881165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.881165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:35.881783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.881783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:35.881783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:35.881783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:35.881783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:35.881783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:35.881783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.881783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:35.881783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:35.881783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.881783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:35.881783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:35.881783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.881783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:35.881783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:35.881783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:35.882064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:35.882064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:35.882064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.882064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:35.882064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:35.882064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:35.882064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:35.882064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:35.882064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:35.882064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:35.882064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:35.882064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:35.882064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:35.882064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:35.882064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:35.882064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:35.882064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:35.882064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:35.882064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:35.882364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:35.882364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:35.882364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:38.755171 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:38.755626 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:38.755626 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:38.756227 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:38.784223 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:38.784454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:38.784988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:38.784988 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:38.784988 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:38.784988 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:38.784988 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:38.784988 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:38.784988 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:38.784988 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:38.785762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:38.785762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:38.785762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:38.785762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:38.785762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:38.785762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:38.785762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:38.785762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:38.785762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:38.785762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:38.785762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:38.785762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:38.786024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:38.786024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:38.786024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:38.786024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:38.786024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:38.786024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:38.786024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:38.786024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:38.786024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:38.786024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:38.786024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:38.786024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:38.786024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:38.786306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:38.786306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:38.786681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:38.786681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:38.786681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:38.786681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:38.786681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:38.786681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:38.786681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:38.786681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:38.786681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:38.786681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:38.786681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:38.786681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:38.786681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:38.786681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:38.786681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:38.786681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:38.787570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:38.787570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:38.787570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:38.787570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:38.787570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:38.787570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:38.787570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:38.787570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:38.787570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:38.787570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:38.787570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:38.787570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:38.787570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:38.787570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:38.787570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:38.787570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:38.787570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:38.787570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:38.787570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:38.787918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:38.787918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:38.787918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:40.838035 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:40.838543 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:40.838599 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:40.839446 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:40.839635 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:40.839749 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:40.839871 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:40.839949 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:40.840053 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:40.840153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.840646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:40.840646 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:40.840646 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:40.840646 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:40.840646 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:40.840646 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:40.840646 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.840646 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:40.841376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.841376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:40.841376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:40.841376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:40.841376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.841376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:40.841376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:40.841376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.841376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:40.841376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:40.841376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:40.841376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.841599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:40.841599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:40.841599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:40.841599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:40.841599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:40.841599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:40.841599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:40.841599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:40.841599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:40.841599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:40.841599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:40.841599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:40.841599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:40.841853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.841853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:40.842265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.842265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:40.842265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:40.842265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:40.842265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:40.842265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:40.842265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.842265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:40.842265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:40.842265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.842265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:40.842265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:40.842265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.842265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:40.842265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:40.842265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.842720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:40.842720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:40.842720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.842720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:40.842720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:40.842720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:40.842720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.842720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:40.842720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:40.842720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:40.842720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:40.842720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:40.842720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:40.842720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:40.842720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:40.842720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:40.842720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:40.842720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:40.842720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:40.843178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:40.843178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.843178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:40.848823 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:40.849083 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:40.849189 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:40.849295 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:37:40.849719 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:40.849848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.850264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:40.850264 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:40.850264 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:40.850264 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:40.850264 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:40.850264 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:40.850264 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.850264 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:40.850918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.850918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:40.850918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:40.850918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:40.850918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.850918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:40.850918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:40.850918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:40.850918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.850918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:40.850918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:40.850918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:40.850918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:40.850918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:40.850918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:40.851191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:40.851191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:40.851191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:40.851191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:40.851191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:40.851191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:40.851191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:40.851191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.851191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:40.851494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.851494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:40.851494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:40.851494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:40.851494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.851494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:40.851494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:40.851494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:37:40.851494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:40.851494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:40.851494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.851494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:40.851816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:40.851816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:40.851816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.851816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:40.851816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:40.851816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:40.851816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:40.851816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:40.851816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:40.851816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:40.851816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:40.851816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:40.851816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:40.851816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:40.852056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:40.852056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:40.852056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.852056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:40.852213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.852213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:40.852213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:40.852213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:40.852213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:40.852213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:40.852213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.852213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:40.852213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:40.852213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.852213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:40.852213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:40.852213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.852213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:40.852213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:40.852213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.852499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:40.852499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:40.852499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:37:40.852499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:40.852499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:40.852499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.852499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:40.852499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:40.852499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:40.852499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.852499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:40.852499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:40.852499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:40.852499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:40.852499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:40.852499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:40.852499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:40.852499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:40.852844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:40.852844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:40.852844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:40.852844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:40.852844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:40.852844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.852844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:40.855244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:40.855531 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:40.855606 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:40.855714 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:40.856134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:40.856367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.856866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:40.856866 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:40.856866 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:40.856866 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:40.856866 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:40.856866 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:40.856866 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.856866 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:40.857434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.857434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:40.857434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:40.857434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:40.857434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.857434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:40.857434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:40.857434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:40.857434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.857434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:40.857434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:40.857434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:40.857434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:40.857434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:40.857434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:40.857811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:40.857811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:40.857811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:40.857811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:40.857811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:40.857811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:40.857811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:40.857811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.857811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:40.858079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.858079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:40.858079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:40.858079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:40.858079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.858079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:40.858079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:40.858079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:40.858079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:40.858079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.858079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:40.858369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:40.858369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:40.858369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.858369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:40.858369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:40.858369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:40.858369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:40.858369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:40.858369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:40.858369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:40.858369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:40.858369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:40.858369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:40.858369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:40.858640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:40.858640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:40.858640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.858640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:40.858808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.858808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:40.858808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:40.858808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:40.858808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:40.858808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:40.858808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.858808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:40.858808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:40.858808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.858808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:40.858808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:40.858808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.858808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:40.858808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:40.858808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.858808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:40.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:40.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:40.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:40.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:40.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:40.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:40.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:40.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:40.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:40.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:40.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:40.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:40.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:40.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:40.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:40.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:40.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:40.859470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:40.859470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:40.859470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.859470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:40.878518 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:40.878997 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:40.879165 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:40.879548 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:40.880050 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:40.880407 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:40.880654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.881126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:40.881126 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:40.881126 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:40.881126 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:40.881126 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:40.881126 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:40.881126 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.881126 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:40.881853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.881853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:40.881853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:40.881853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:40.881853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.881853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:40.881853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:40.881853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.881853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:40.881853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:40.881853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:40.881853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.882327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:40.882327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:40.882327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:40.882327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:40.882327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:40.882327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:40.882327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:40.882327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:40.882327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:40.882327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:40.882327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:40.882327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:40.882327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:40.882735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.882735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:40.882735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.882735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:40.882735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:40.882735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:40.882735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:40.882735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:40.882735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.882735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:40.882735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:40.882735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.882735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:40.882735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:40.882735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.883189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:40.883189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:40.883189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:40.883189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:40.883189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:40.883189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.883189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:40.883189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:40.883189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:40.883189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:40.883189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:40.883189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:40.883189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:40.883189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:40.883189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:40.883189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:40.883189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:40.883189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:40.883189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:40.883762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:40.883762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:40.883762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:40.883762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:40.883762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:40.883762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:43.755588 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:43.756074 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:43.756074 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:43.756578 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:43.785991 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:43.786356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:43.787026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:43.787026 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:43.787026 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:43.787026 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:43.787026 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:43.787026 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:43.787026 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:43.787026 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:43.787984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:43.787984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:43.787984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:43.787984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:43.787984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:43.787984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:43.787984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:43.787984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:43.787984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:43.787984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:43.787984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:43.787984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:43.788525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:43.788525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:43.788525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:43.788525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:43.788525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:43.788525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:43.788525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:43.788525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:43.788525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:43.788525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:43.788525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:43.788525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:43.788525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:43.789057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:43.789057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:43.789057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:43.789057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:43.789057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:43.789057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:43.789057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:43.789057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:43.789057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:43.789057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:43.789057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:43.789057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:43.789057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:43.789057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:43.789057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:43.789638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:43.789638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:43.789638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:43.789638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:43.789638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:43.789638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:43.789638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:43.789638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:43.789638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:43.789638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:43.789638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:43.789638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:43.789638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:43.789638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:43.789638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:43.789638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:43.789638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:43.789638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:43.789638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:43.790380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:43.790380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:43.790380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:43.790380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:43.790380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:43.790380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:45.838550 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:45.839123 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:45.839123 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:45.840087 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:45.840373 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:45.840565 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:45.840741 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:45.840910 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:45.841056 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:45.841206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.841853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:45.841853 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:45.841853 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:45.841853 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:45.841853 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:45.841853 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:45.841853 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.841853 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:45.842616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.842616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:45.842616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:45.842616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:45.842616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.842616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:45.842616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:45.842616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.842616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:45.842616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:45.842616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:45.842616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.842906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:45.842906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:45.842906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:45.842906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:45.842906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:45.842906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:45.842906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:45.842906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:45.842906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:45.842906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:45.842906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:45.842906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:45.842906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:45.843144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.843144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:45.843345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.843345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:45.843345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:45.843345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:45.843345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:45.843345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:45.843345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.843345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:45.843345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:45.843345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.843345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:45.843345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:45.843345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.843345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:45.843345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:45.843345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.843667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:45.843667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:45.843667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.843667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:45.843667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:45.843667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:45.843667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.843667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:45.843667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:45.843667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:45.843667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:45.843667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:45.843667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:45.843667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:45.843667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:45.843667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:45.843667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:45.843667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:45.843667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:45.844093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:45.844093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.844093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:45.849202 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:45.849443 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:45.849547 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:45.849656 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:37:45.849974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:45.850135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.850555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:45.850555 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:45.850555 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:45.850555 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:45.850555 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:45.850555 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:45.850555 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.850555 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:45.851121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.851121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:45.851121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:45.851121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:45.851121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.851121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:45.851121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:45.851121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:45.851121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.851121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:45.851121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:45.851121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:45.851121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:45.851121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:45.851121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:45.851503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:45.851503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:45.851503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:45.851503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:45.851503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:45.851503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:45.851503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:45.851503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.851503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:45.851714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.851714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:45.851714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:45.851714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:45.851714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.851714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:45.851714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:45.851714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:37:45.851714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:45.851714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:45.851714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.851714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:45.851950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:45.851950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:45.851950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.851950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:45.851950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:45.851950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:45.851950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:45.851950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:45.851950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:45.851950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:45.851950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:45.851950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:45.851950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:45.851950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:45.852215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:45.852215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:45.852215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.852215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:45.852472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.852472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:45.852472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:45.852472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:45.852472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:45.852472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:45.852472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.852472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:45.852472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:45.852472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.852472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:45.852472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:45.852472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.852472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:45.852472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:45.852472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.852777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:45.852777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:45.852777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:37:45.852777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:45.852777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:45.852777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.852777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:45.852777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:45.852777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:45.852777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.852777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:45.852777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:45.852777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:45.852777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:45.852777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:45.852777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:45.852777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:45.852777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:45.853093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:45.853093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:45.853093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:45.853093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:45.853093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:45.853093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.853093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:45.855640 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:45.855869 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:45.855968 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:45.856061 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:45.856327 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:45.856492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.856864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:45.856864 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:45.856864 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:45.856864 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:45.856864 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:45.856864 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:45.856864 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.856864 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:45.857386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.857386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:45.857386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:45.857386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:45.857386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.857386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:45.857386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:45.857386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:45.857386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.857386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:45.857386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:45.857386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:45.857386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:45.857386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:45.857386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:45.857698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:45.857698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:45.857698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:45.857698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:45.857698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:45.857698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:45.857698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:45.857698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.857698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:45.857983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.857983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:45.857983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:45.857983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:45.857983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.857983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:45.857983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:45.857983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:45.857983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:45.857983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.857983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:45.858219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:45.858219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:45.858219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.858219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:45.858219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:45.858219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:45.858219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:45.858219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:45.858219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:45.858219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:45.858219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:45.858219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:45.858219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:45.858219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:45.858462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:45.858462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:45.858462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.858462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:45.858806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.858806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:45.858806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:45.858806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:45.858806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:45.858806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:45.858806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.858806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:45.858806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:45.858806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.858806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:45.858806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:45.858806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.858806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:45.858806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:45.858806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.858806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:45.859201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:45.859201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:45.859201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:45.859201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.859201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:45.859201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:45.859201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:45.859201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.859201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:45.859201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:45.859201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:45.859201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:45.859201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:45.859201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:45.859201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:45.859201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:45.859201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:45.859201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:45.859201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:45.859581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:45.859581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:45.859581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.859581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:45.878941 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:45.879200 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:45.879312 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:45.879450 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:45.879750 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:45.880002 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:45.880245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.880687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:45.880687 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:45.880687 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:45.880687 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:45.880687 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:45.880687 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:45.880687 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.880687 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:45.881486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.881486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:45.881486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:45.881486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:45.881486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.881486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:45.881486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:45.881486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.881486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:45.881486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:45.881486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:45.881486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.881750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:45.881750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:45.881750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:45.881750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:45.881750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:45.881750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:45.881750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:45.881750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:45.881750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:45.881750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:45.881750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:45.881750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:45.881750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:45.882009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.882009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:45.882165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.882165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:45.882165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:45.882165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:45.882165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:45.882165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:45.882165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.882165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:45.882165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:45.882165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.882165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:45.882165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:45.882165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.882165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:45.882165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:45.882165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:45.882528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:45.882528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:45.882528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.882528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:45.882528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:45.882528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:45.882528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:45.882528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:45.882528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:45.882528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:45.882528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:45.882528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:45.882528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:45.882528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:45.882528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:45.882528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:45.882528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:45.882528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:45.882528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:45.882885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:45.882885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:45.882885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:48.756506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:48.758575 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:48.758575 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:48.758575 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:48.788604 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:48.788780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:48.789359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:48.789359 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:48.789359 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:48.789359 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:48.789359 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:48.789359 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:48.789359 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:48.789359 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:48.790053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:48.790053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:48.790053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:48.790053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:48.790053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:48.790053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:48.790053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:48.790053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:48.790053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:48.790053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:48.790053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:48.790053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:48.790526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:48.790526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:48.790526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:48.790526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:48.790526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:48.790526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:48.790526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:48.790526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:48.790526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:48.790526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:48.790526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:48.790526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:48.790526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:48.791136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:48.791136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:48.791136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:48.791136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:48.791136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:48.791136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:48.791136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:48.791136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:48.791136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:48.791136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:48.791136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:48.791136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:48.791136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:48.791136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:48.791136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:48.791766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:48.791766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:48.791766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:48.791766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:48.791766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:48.791766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:48.791766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:48.791766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:48.791766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:48.791766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:48.791766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:48.791766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:48.791766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:48.791766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:48.791766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:48.791766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:48.791766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:48.791766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:48.791766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:48.792492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:48.792492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:48.792492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:48.792492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:48.792492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:48.792492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:50.839527 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:50.840000 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:50.840129 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:50.841096 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:50.841270 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:50.841413 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:50.841520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:50.841648 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:50.841746 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:50.841847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.842306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:50.842306 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:50.842306 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:50.842306 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:50.842306 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:50.842306 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:50.842306 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.842306 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:50.843223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.843223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:50.843223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:50.843223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:50.843223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.843223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:50.843223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:50.843223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.843223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:50.843223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:50.843223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:50.843223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.843490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:50.843490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:50.843490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:50.843490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:50.843490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:50.843490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:50.843490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:50.843490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:50.843490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:50.843490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:50.843490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:50.843490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:50.843490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:50.843711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.843711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:50.844361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.844361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:50.844361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:50.844361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:50.844361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:50.844361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:50.844361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.844361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:50.844361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:50.844361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.844361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:50.844361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:50.844361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.844361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:50.844361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:50.844361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.844673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:50.844673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:50.844673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.844673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:50.844673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:50.844673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:50.844673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.844673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:50.844673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:50.844673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:50.844673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:50.844673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:50.844673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:50.844673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:50.844673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:50.844673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:50.844673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:50.844673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:50.844673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:50.844989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:50.844989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.844989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:50.849516 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:50.849896 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:50.850030 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:50.850166 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:37:50.850502 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:50.850706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.851148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:50.851148 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:50.851148 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:50.851148 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:50.851148 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:50.851148 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:50.851148 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.851148 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:50.851956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.851956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:50.851956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:50.851956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:50.851956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.851956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:50.851956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:50.851956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:50.851956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.851956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:50.851956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:50.851956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:50.851956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:50.851956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:50.851956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:50.852425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:50.852425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:50.852425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:50.852425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:50.852425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:50.852425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:50.852425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:50.852425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.852425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:50.852889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.852889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:50.852889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:50.852889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:50.852889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.852889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:50.852889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:50.852889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:37:50.852889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:50.852889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:50.852889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.852889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:50.853231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:50.853231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:50.853231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.853231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:50.853231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:50.853231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:50.853231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:50.853231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:50.853231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:50.853231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:50.853231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:50.853231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:50.853231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:50.853231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:50.854003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:50.854003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:50.854003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.854003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:50.854117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.854117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:50.854117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:50.854117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:50.854117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:50.854117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:50.854117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.854117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:50.854117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:50.854117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.854117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:50.854117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:50.854117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.854117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:50.854117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:50.854117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.854398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:50.854398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:50.854398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:37:50.854398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:50.854398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:50.854398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.854398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:50.854398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:50.854398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:50.854398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.854398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:50.854398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:50.854398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:50.854398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:50.854398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:50.854398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:50.854398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:50.854398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:50.854713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:50.854713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:50.854713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:50.854713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:50.854713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:50.854713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.854713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:50.855988 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:50.856323 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:50.856455 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:50.856626 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:50.856937 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:50.857151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.857579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:50.857579 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:50.857579 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:50.857579 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:50.857579 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:50.857579 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:50.857579 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.857579 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:50.858388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.858388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:50.858388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:50.858388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:50.858388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.858388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:50.858388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:50.858388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:50.858388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.858388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:50.858388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:50.858388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:50.858388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:50.858388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:50.858388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:50.858689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:50.858689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:50.858689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:50.858689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:50.858689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:50.858689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:50.858689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:50.858689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.858689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:50.859310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.859310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:50.859310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:50.859310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:50.859310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.859310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:50.859310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:50.859310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:50.859310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:50.859310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.859310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:50.859533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:50.859533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:50.859533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.859533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:50.859533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:50.859533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:50.859533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:50.859533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:50.859533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:50.859533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:50.859533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:50.859533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:50.859533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:50.859533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:50.859779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:50.859779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:50.859779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.859779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:50.860453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.860453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:50.860453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:50.860453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:50.860453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:50.860453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:50.860453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.860453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:50.860453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:50.860453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.860453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:50.860453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:50.860453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.860453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:50.860453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:50.860453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.860453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:50.860792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:50.860792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:50.860792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:50.860792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.860792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:50.860792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:50.860792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:50.860792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.860792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:50.860792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:50.860792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:50.860792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:50.860792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:50.860792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:50.860792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:50.860792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:50.860792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:50.860792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:50.860792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:50.861107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:50.861107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:50.861107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.861107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:50.879312 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:50.879730 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:50.879846 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:50.880010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:50.880373 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:50.880671 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:50.880919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.881480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:50.881480 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:50.881480 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:50.881480 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:50.881480 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:50.881480 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:50.881480 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.881480 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:50.882294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.882294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:50.882294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:50.882294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:50.882294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.882294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:50.882294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:50.882294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.882294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:50.882294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:50.882294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:50.882294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.882606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:50.882606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:50.882606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:50.882606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:50.882606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:50.882606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:50.882606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:50.882606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:50.882606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:50.882606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:50.882606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:50.882606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:50.882606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:50.882818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.882818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:50.883210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.883210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:50.883210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:50.883210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:50.883210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:50.883210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:50.883210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.883210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:50.883210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:50.883210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.883210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:50.883210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:50.883210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.883210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:50.883210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:50.883210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:50.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:50.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:50.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:50.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:50.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:50.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:50.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:50.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:50.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:50.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:50.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:50.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:50.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:50.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:50.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:50.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:50.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:50.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:50.883906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:50.883906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:50.883906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:53.757364 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:53.757873 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:53.757984 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:53.758882 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:53.769916 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=241 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:37:53.801923 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:53.802189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:53.802803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:53.802803 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:53.802803 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:53.802803 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:53.802803 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:53.802803 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:53.802803 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:53.802803 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:53.803902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:53.803902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:53.803902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:53.803902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:53.803902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:53.803902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:53.803902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:53.803902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:53.803902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:53.803902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:53.803902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:53.803902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:53.804236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:53.804236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:53.804236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:53.804236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:53.804236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:53.804236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:53.804236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:53.804236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:53.804236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:53.804236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:53.804236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:53.804236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:53.804236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:53.804571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:53.804571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:53.805088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:53.805088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:53.805088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:53.805088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:53.805088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:53.805088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:53.805088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:53.805088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:53.805088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:53.805088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:53.805088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:53.805088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:53.805088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:53.805088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:53.805088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:53.805088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:53.805498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:53.805498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:53.805498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:53.805498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:53.805498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:53.805498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:53.805498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:53.805498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:53.805498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:53.805498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:53.805498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:53.805498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:53.805498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:53.805498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:53.805498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:53.805498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:53.805498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:53.805498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:53.805498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:53.805868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:53.805868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:53.805868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:55.839980 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:55.840418 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:55.840577 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:55.842016 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:55.842308 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:55.842436 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:55.842860 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:55.842860 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:55.842860 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:55.842860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.843487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:55.843487 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:55.843487 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:55.843487 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:55.843487 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:55.843487 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:55.843487 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.843487 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:55.844367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.844367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:55.844367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:55.844367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:55.844367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.844367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:55.844367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:55.844367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.844367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:55.844367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:55.844367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:55.844367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.844955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:55.844955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:55.844955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:55.844955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:55.844955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:55.844955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:55.844955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:55.844955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:55.844955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:55.844955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:55.844955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:55.844955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:55.844955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:55.845475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.845475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:55.845475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.845475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:55.845475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:55.845475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:55.845475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:55.845475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:55.845475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.845475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:55.845475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:55.845475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.845475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:55.845475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:55.845475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.846026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:55.846026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:55.846026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.846026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:37:55.846026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:55.846026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.846026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:55.846026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:55.846026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:55.846026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.846026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:55.846026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:55.846026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:55.846026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:55.846026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:55.846026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:55.846026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:55.846026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:55.846026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:55.846432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:55.846432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:55.846432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:55.846432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:55.846432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.846432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:55.849812 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:55.850058 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:55.850162 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:55.850332 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:37:55.853736 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:55.853938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.854460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:55.854460 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:55.854460 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:55.854460 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:55.854460 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:55.854460 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:55.854460 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.854460 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:55.855124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.855124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:55.855124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:55.855124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:55.855124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.855124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:55.855124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:55.855124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:55.855124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.855124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:55.855124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:55.855124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:55.855124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:55.855124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:55.855124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:55.855476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:55.855476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:55.855476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:55.855476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:55.855476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:55.855476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:55.855476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:55.855476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.855476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:55.855925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.855925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:55.855925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:55.855925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:55.855925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.855925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:55.855925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:55.855925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:37:55.855925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:55.855925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:55.855925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.855925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:55.856163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:55.856163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:55.856163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.856163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:55.856163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:55.856163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:55.856163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:55.856163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:55.856163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:55.856163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:55.856163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:55.856163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:55.856163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:55.856163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:55.856444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:55.856444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:55.856444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.856444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:55.856912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.856912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:55.856912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:55.856912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:55.856912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:55.856912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:55.856912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.856912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:55.856912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:55.856912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.856912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:55.856912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:55.856912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.856912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:55.856912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:55.856912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.857183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:37:55.857183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:37:55.857183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:37:55.857183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:55.857183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:55.857183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.857183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:55.857183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:55.857183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:55.857183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.857183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:55.857183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:55.857183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:55.857183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:55.857183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:55.857183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:55.857183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:55.857183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:55.857535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:55.857535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:55.857535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:55.857535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:55.857535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:55.857535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.857535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:55.857535 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:55.857535 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:55.857535 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:55.857738 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:55.857989 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:55.858267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.858701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:55.858701 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:55.858701 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:55.858701 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:55.858701 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:55.858701 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:55.858701 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.858701 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:55.859244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.859244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:37:55.859244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:55.859244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:55.859244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.859244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:55.859244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:55.859244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:37:55.859244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.859244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:37:55.859244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:55.859244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:37:55.859244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:55.859244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:37:55.859244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:37:55.859527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:55.859527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:37:55.859527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:55.859527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:37:55.859527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:37:55.859527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:55.859527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:37:55.859527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.859527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:37:55.860046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.860046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:55.860046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:55.860046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:55.860046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.860046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:55.860046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:55.860046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:55.860046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:55.860046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.860046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:55.860313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:55.860313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:55.860313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.860313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:55.860313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:55.860313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:55.860313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:55.860313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:55.860313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:55.860313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:55.860313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:55.860313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:55.860313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:55.860313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:55.860550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:55.860550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:55.860550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.860550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:55.861046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.861046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:55.861046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:55.861046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:55.861046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:55.861046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:55.861046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.861046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:55.861046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:55.861046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.861046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:55.861046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:55.861046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.861046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:55.861046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:55.861046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.861046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:37:55.861388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:37:55.861388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:37:55.861388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:37:55.861388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.861388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:55.861388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:55.861388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:55.861388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.861388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:55.861388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:55.861388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:55.861388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:55.861388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:55.861388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:55.861388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:55.861388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:55.861388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:55.861388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:55.861388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:55.861732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:55.861732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:55.861732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.861732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:55.879721 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:55.879982 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:55.880113 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:55.880406 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:55.880863 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:55.881302 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:55.881407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.884107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:55.884107 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:55.884107 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:55.884107 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:55.884107 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:55.884107 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:55.884107 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.884107 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:55.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:55.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:55.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:55.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:55.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:55.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:55.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:55.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:55.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.885424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:55.885424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:55.885424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:55.885424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:55.885424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:55.885424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:55.885424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:55.885424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:55.885424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:55.885424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:55.885424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:55.885424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:55.885424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:55.885911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.885911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:55.886004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.886004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:55.886004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:55.886004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:55.886004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:55.886004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:55.886004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.886004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:55.886004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:55.886004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.886004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:55.886004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:55.886004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.886004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:55.886004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:55.886004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:55.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:37:55.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:55.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:55.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:55.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:55.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:55.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:55.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:55.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:55.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:55.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:55.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:55.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:55.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:55.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:55.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:55.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:55.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:55.886685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:55.886685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:55.886685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:37:58.757894 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:37:58.758461 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:37:58.758461 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:37:58.759315 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:37:58.794884 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:37:58.795629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:58.795707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:37:58.795707 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:58.795707 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:37:58.795707 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:37:58.795707 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:58.795707 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:37:58.795707 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:58.795707 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:37:58.796489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:58.796489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:37:58.796489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:58.796489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:37:58.796489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:58.796489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:58.796489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:58.796489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:58.796489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:58.796489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:58.796489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:37:58.796489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:58.796883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:37:58.796883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:58.796883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:37:58.796883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:58.796883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:37:58.796883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:37:58.796883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:58.796883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:37:58.796883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:58.796883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:37:58.796883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:37:58.796883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:58.796883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:37:58.797235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:58.797235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:37:58.797322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:58.797322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:37:58.797322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:37:58.797322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:37:58.797322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:37:58.797322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:37:58.797322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:58.797322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:37:58.797322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:37:58.797322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:58.797322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:37:58.797322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:37:58.797322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:37:58.797322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:37:58.797322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:37:58.797322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:37:58.797777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:37:58.797777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:37:58.797777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:58.797777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:58.797777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:37:58.797777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:37:58.797777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:37:58.797777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:37:58.797777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:37:58.797777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:37:58.797777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:37:58.797777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:37:58.797777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:37:58.797777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:37:58.797777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:37:58.797777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:37:58.797777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:37:58.797777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:37:58.797777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:37:58.798277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:37:58.798277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:37:58.798277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:00.840440 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:00.841057 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:00.841057 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:00.841569 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:00.841746 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:00.841883 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:00.842012 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:00.842140 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:00.842279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:00.842409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.842941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:00.842941 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:00.842941 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:00.842941 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:00.842941 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:00.842941 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:00.842941 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.842941 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:00.843583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.843583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:00.843583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:00.843583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:00.843583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.843583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:00.843583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:00.843583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.843583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:00.843583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:00.843583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:00.843583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.843999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:00.843999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:00.843999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:00.843999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:00.843999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:00.843999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:00.843999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:00.843999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:00.843999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:00.843999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:00.843999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:00.843999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:00.843999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:00.844438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.844438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:00.844720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.844720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:00.844720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:00.844720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:00.844720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:00.844720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:00.844720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.844720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:00.844720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:00.844720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.844720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:00.844720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:00.844720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.844720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:00.844720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:00.844720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.845127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:00.845127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:00.845127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.845127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:00.845127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:00.845127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:00.845127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.845127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:00.845127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:00.845127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:00.845127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:00.845127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:00.845127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:00.845127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:00.845127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:00.845127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:00.845127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:00.845127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:00.845127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:00.845478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:00.845478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.845478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:00.850172 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:00.850412 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:00.850527 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:00.850672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:38:00.850995 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:00.851176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.852323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:00.852323 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:00.852323 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:00.852323 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:00.852323 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:00.852323 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:00.852323 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.852323 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:00.852684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.852684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:00.852684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:00.852684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:00.852684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.852684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:00.852684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:00.852684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:00.852684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.852684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:00.852684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:00.852684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:00.852684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:00.852684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:00.852684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:00.853136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:00.853136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:00.853136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:00.853136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:00.853136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:00.853136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:00.853136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:00.853136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.853136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:00.853408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.853408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:00.853408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:00.853408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:00.853408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.853408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:00.853408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:00.853408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:38:00.853408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:00.853408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:00.853408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.853408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:00.853754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:00.853754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:00.853754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.853754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:00.853754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:00.853754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:00.853754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:00.853754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:00.853754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:00.853754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:00.853754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:00.853754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:00.853754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:00.853754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:00.854157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:00.854157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:00.854157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.854157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:00.854296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.854296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:00.854296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:00.854296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:00.854296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:00.854296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:00.854296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.854296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:00.854296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:00.854296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.854296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:00.854296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:00.854296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.854296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:00.854296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:00.854296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.854785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:00.854785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:00.854785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:38:00.854785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:00.854785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:00.854785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.854785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:00.854785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:00.854785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:00.854785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.854785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:00.854785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:00.854785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:00.854785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:00.854785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:00.854785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:00.854785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:00.854785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:00.855274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:00.855274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:00.855274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:00.855274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:00.855274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:00.855274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.855274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:00.858318 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:00.858499 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:00.858602 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:00.858744 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:00.859056 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:00.859326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.862012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:00.862012 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:00.862012 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:00.862012 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:00.862012 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:00.862012 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:00.862012 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.862012 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:00.862554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.862554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:00.862554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:00.862554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:00.862554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.862554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:00.862554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:00.862554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:00.862554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.862554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:00.862554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:00.862554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:00.862554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:00.862554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:00.862554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:00.863007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:00.863007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:00.863007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:00.863007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:00.863007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:00.863007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:00.863007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:00.863007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.863007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:00.863270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.863270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:00.863270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:00.863270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:00.863270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.863270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:00.863270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:00.863270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:00.863270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:00.863270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.863270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:00.863609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:00.863609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:00.863609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.863609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:00.863609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:00.863609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:00.863609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:00.863609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:00.863609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:00.863609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:00.863609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:00.863609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:00.863609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:00.863609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:00.863984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:00.863984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:00.863984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.863984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:00.863984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.863984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:00.863984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:00.863984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:00.863984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:00.863984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:00.863984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.863984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:00.863984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:00.863984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.864404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:00.864404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:00.864404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.864404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:00.864404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:00.864404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.864404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:00.864404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:00.864404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:00.864404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:00.864404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.864404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:00.864404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:00.864404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:00.864404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.864404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:00.864404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:00.864404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:00.864404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:00.865016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:00.865016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:00.865016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:00.865016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:00.865016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:00.865016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:00.865016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:00.865016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:00.865016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:00.865016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.865016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:00.880086 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:00.880341 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:00.880425 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:00.880564 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:00.880903 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:00.881136 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:00.881366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.881910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:00.881910 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:00.881910 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:00.881910 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:00.881910 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:00.881910 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:00.881910 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.881910 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:00.882572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.882572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:00.882572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:00.882572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:00.882572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.882572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:00.882572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:00.882572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.882572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:00.882572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:00.882572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:00.882572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.882939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:00.882939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:00.882939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:00.882939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:00.882939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:00.882939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:00.882939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:00.882939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:00.882939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:00.882939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:00.882939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:00.882939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:00.882939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:00.883373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.883373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:00.883501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.883501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:00.883501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:00.883501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:00.883501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:00.883501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:00.883501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.883501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:00.883501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:00.883501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.883501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:00.883501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:00.883501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.883501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:00.883501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:00.883501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:00.883966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:00.883966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:00.883966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.883966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:00.883966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:00.883966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:00.883966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:00.883966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:00.883966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:00.883966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:00.883966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:00.883966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:00.883966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:00.883966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:00.883966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:00.883966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:00.883966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:00.883966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:00.883966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:00.884486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:00.884486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:00.884486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:03.758416 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:03.758928 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:03.758928 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:03.759669 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:03.795411 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:03.795616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:03.796088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:03.796088 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:03.796088 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:03.796088 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:03.796088 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:03.796088 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:03.796088 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:03.796088 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:03.796788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:03.796788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:03.796788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:03.796788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:03.796788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:03.796788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:03.796788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:03.796788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:03.796788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:03.796788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:03.796788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:03.796788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:03.797111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:03.797111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:03.797111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:03.797111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:03.797111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:03.797111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:03.797111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:03.797111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:03.797111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:03.797111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:03.797111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:03.797111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:03.797111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:03.797461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:03.797461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:03.797543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:03.797543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:03.797543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:03.797543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:03.797543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:03.797543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:03.797543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:03.797543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:03.797543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:03.797543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:03.797543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:03.797543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:03.797543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:03.797543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:03.797543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:03.797543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:03.797970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:03.797970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:03.797970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:03.797970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:03.797970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:03.797970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:03.797970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:03.797970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:03.797970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:03.797970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:03.797970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:03.797970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:03.797970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:03.797970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:03.797970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:03.797970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:03.797970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:03.797970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:03.797970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:03.798415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:03.798415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:03.798415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:05.840814 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:05.841356 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:05.841356 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:05.841933 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:05.842132 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:05.842272 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:05.842398 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:05.842526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:05.842678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:05.842863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.843319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:05.843319 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:05.843319 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:05.843319 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:05.843319 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:05.843319 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:05.843319 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.843319 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:05.843987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.843987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:05.843987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:05.843987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:05.843987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.843987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:05.843987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:05.843987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.843987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:05.843987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:05.843987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:05.843987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.844437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:05.844437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:05.844437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:05.844437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:05.844437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:05.844437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:05.844437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:05.844437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:05.844437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:05.844437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:05.844437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:05.844437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:05.844437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:05.844842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.844842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:05.844842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.844842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:05.844842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:05.844842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:05.844842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:05.844842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:05.844842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.844842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:05.844842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:05.844842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.844842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:05.844842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:05.844842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.845294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:05.845294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:05.845294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.845294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:05.845294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:05.845294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.845294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:05.845294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:05.845294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:05.845294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.845294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:05.845294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:05.845294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:05.845294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:05.845294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:05.845294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:05.845294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:05.845294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:05.845294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:05.845839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:05.845839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:05.845839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:05.845839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:05.845839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.845839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:05.850420 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:05.850578 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:05.850698 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:05.850790 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:38:05.851088 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:05.851233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.851667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:05.851667 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:05.851667 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:05.851667 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:05.851667 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:05.851667 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:05.851667 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.851667 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:05.852157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.852157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:05.852157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:05.852157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:05.852157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.852157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:05.852157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:05.852157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:05.852157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.852157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:05.852157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:05.852157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:05.852157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:05.852157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:05.852157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:05.852576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:05.852576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:05.852576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:05.852576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:05.852576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:05.852576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:05.852576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:05.852576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.852576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:05.852852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.852852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:05.852852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:05.852852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:05.852852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.852852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:05.852852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:05.852852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:38:05.852852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:05.852852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:05.852852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.852852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:05.853204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:05.853204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:05.853204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.853204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:05.853204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:05.853204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:05.853204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:05.853204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:05.853204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:05.853204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:05.853204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:05.853204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:05.853204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:05.853204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:05.853605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:05.853605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:05.853605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.853605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:05.853605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.853605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:05.853605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:05.853605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:05.853605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:05.853605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:05.853605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.853605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:05.853605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:05.853605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.854018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:05.854018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:05.854018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.854018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:05.854018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:05.854018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.854018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:05.854018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:05.854018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:38:05.854018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:05.854018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:05.854018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.854018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:05.854018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:05.854018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:05.854018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.854018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:05.854018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:05.854018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:05.854578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:05.854578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:05.854578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:05.854578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:05.854578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:05.854578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:05.854578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:05.854578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:05.854578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:05.854578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:05.854578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.854578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:05.858529 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:05.858757 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:05.858854 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:05.858956 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:05.859244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:05.859440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.859867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:05.859867 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:05.859867 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:05.859867 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:05.859867 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:05.859867 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:05.859867 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.859867 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:05.860410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.860410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:05.860410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:05.860410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:05.860410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.860410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:05.860410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:05.860410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:05.860410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.860410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:05.860410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:05.860410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:05.860410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:05.860410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:05.860410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:05.860677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:05.860677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:05.860677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:05.860677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:05.860677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:05.860677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:05.860677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:05.860677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.860677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:05.860992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.860992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:05.860992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:05.860992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:05.860992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.860992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:05.860992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:05.860992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:05.860992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:05.860992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.860992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:05.861185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:05.861185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:05.861185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.861185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:05.861185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:05.861185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:05.861185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:05.861185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:05.861185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:05.861185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:05.861185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:05.861185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:05.861185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:05.861185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:05.861487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:05.861487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:05.861487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.861487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:05.861690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.861690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:05.861690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:05.861690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:05.861690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:05.861690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:05.861690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.861690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:05.861690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:05.861690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.861690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:05.861690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:05.861690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.861690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:05.861690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:05.861690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.861690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:05.862199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:05.862199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:05.862199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:05.862199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.862199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:05.862199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:05.862199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:05.862199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.862199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:05.862199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:05.862199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:05.862199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:05.862199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:05.862199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:05.862199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:05.862199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:05.862199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:05.862199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:05.862199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:05.862771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:05.862771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:05.862771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.862771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:05.880364 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:05.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:05.880644 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:05.880763 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:05.881058 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:05.881238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:05.881425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.881815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:05.881815 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:05.881815 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:05.881815 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:05.881815 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:05.881815 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:05.881815 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.881815 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:05.882397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.882397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:05.882397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:05.882397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:05.882397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.882397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:05.882397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:05.882397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.882397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:05.882397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:05.882397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:05.882397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.882761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:05.882761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:05.882761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:05.882761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:05.882761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:05.882761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:05.882761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:05.882761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:05.882761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:05.882761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:05.882761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:05.882761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:05.882761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:05.883122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.883122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:05.883122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.883122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:05.883122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:05.883122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:05.883122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:05.883122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:05.883122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.883122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:05.883122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:05.883122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.883122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:05.883122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:05.883122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.883560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:05.883560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:05.883560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:05.883560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:05.883560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:05.883560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.883560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:05.883560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:05.883560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:05.883560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:05.883560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:05.883560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:05.883560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:05.883560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:05.883560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:05.883560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:05.883560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:05.883560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:05.883560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:05.884166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:05.884166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:05.884166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:05.884166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:05.884166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:05.884166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:08.758860 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:08.759314 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:08.759314 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:08.759920 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:08.788233 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:08.788424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:08.788951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:08.788951 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:08.788951 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:08.788951 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:08.788951 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:08.788951 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:08.788951 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:08.788951 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:08.789631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:08.789631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:08.789631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:08.789631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:08.789631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:08.789631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:08.789631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:08.789631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:08.789631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:08.789631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:08.789631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:08.789631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:08.789909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:08.789909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:08.789909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:08.789909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:08.789909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:08.789909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:08.789909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:08.789909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:08.789909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:08.789909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:08.789909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:08.789909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:08.789909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:08.790123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:08.790123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:08.790346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:08.790346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:08.790346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:08.790346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:08.790346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:08.790346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:08.790346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:08.790346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:08.790346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:08.790346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:08.790346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:08.790346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:08.790346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:08.790346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:08.790346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:08.790346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:08.791044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:08.791044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:08.791044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:08.791044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:08.791044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:08.791044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:08.791044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:08.791044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:08.791044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:08.791044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:08.791044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:08.791044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:08.791044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:08.791044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:08.791044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:08.791044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:08.791044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:08.791044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:08.791044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:08.791696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:08.791696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:08.791696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:10.841215 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:10.843635 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:10.843635 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:10.843635 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:10.843635 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:10.843635 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:10.843789 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:10.843789 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:10.843789 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:10.843789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.843890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:10.843890 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:10.843890 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:10.843890 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:10.843890 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:10.843890 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:10.843890 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.843890 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:10.844723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.844723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:10.844723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:10.844723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:10.844723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.844723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:10.844723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:10.844723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.844723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:10.844723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:10.844723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:10.844723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.847182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:10.847182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:10.847182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:10.847182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:10.847182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:10.847182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:10.847182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:10.847182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:10.847182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:10.847182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:10.847182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:10.847182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:10.847182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:10.847603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.847603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:10.847603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.847603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:10.847603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:10.847603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:10.847603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:10.847603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:10.847603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.847603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:10.847603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:10.847603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.847603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:10.847603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:10.847603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.847987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:10.847987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:10.847987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.847987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:10.847987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:10.847987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.847987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:10.847987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:10.847987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:10.847987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.847987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:10.847987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:10.847987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:10.847987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:10.847987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:10.847987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:10.847987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:10.847987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:10.847987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:10.848350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:10.848350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:10.848350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:10.848350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:10.848350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.848350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:10.851880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:10.851880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:10.851880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:10.851880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:38:10.851880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:10.851880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.852340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:10.852340 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:10.852340 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:10.852340 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:10.852340 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:10.852340 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:10.852340 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.852340 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:10.852842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.852842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:10.852842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:10.852842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:10.852842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.852842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:10.852842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:10.852842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:10.852842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.852842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:10.852842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:10.852842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:10.852842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:10.852842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:10.852842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:10.855190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:10.855190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:10.855190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:10.855190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:10.855190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:10.855190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:10.855190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:10.855190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.855190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:10.855190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.855190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:10.855190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:10.855190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:10.855190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.855535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:10.855535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:10.855535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:38:10.855535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:10.855535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:10.855535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.855535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:10.855535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:10.855535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:10.855535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.855535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:10.855535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:10.855535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:10.855758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:10.855758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:10.855758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:10.855758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:10.855758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:10.855758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:10.855758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:10.855758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:10.855758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:10.855758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:10.855758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.855758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:10.855999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.855999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:10.855999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:10.855999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:10.855999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:10.855999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:10.855999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.855999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:10.855999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:10.855999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.855999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:10.855999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:10.855999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.855999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:10.855999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:10.855999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.856246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:10.856246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:10.856246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:38:10.856246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:10.856246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:10.856246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.856246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:10.856246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:10.856246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:10.856246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.856246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:10.856246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:10.856246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:10.856246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:10.856246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:10.856246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:10.856246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:10.856246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:10.856539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:10.856539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:10.856539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:10.856539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:10.856539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:10.856539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.856539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:10.858849 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:10.859059 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:10.859141 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:10.859237 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:10.859607 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:10.859803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.860201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:10.860201 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:10.860201 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:10.860201 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:10.860201 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:10.860201 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:10.860201 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.860201 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:10.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:10.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:10.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:10.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:10.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:10.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:10.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:10.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:10.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:10.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:10.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:10.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:10.861150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:10.861150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:10.861150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:10.861150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:10.861150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:10.861150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:10.861150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:10.861150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.861150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:10.861530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.861530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:10.861530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:10.861530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:10.861530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.861530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:10.861530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:10.861530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:10.861530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:10.861530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.861530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:10.861734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:10.861734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:10.861734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.861734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:10.861734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:10.861734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:10.861734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:10.861734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:10.861734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:10.861734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:10.861734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:10.861734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:10.861734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:10.861734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:10.861945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:10.861945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:10.861945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.861945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:10.862212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.862212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:10.862212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:10.862212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:10.862212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:10.862212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:10.862212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.862212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:10.862212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:10.862212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.862212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:10.862212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:10.862212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.862212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:10.862212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:10.862212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.862212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:10.862682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:10.862682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:10.862682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:10.862682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.862682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:10.862682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:10.862682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:10.862682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.862682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:10.862682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:10.862682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:10.862682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:10.862682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:10.862682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:10.862682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:10.862682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:10.862682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:10.862682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:10.862682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:10.862989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:10.862989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:10.862989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.862989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:10.881531 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:10.881531 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:10.881531 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:10.881648 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:10.882015 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:10.882236 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:10.882540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.882930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:10.882930 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:10.882930 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:10.882930 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:10.882930 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:10.882930 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:10.882930 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.882930 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:10.883584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.883584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:10.883584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:10.883584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:10.883584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.883584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:10.883584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:10.883584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.883584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:10.883584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:10.883584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:10.883584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.883838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:10.883838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:10.883838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:10.883838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:10.883838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:10.883838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:10.883838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:10.883838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:10.883838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:10.883838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:10.883838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:10.883838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:10.883838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:10.884056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.884056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:10.884295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.884295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:10.884295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:10.884295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:10.884295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:10.884295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:10.884295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.884295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:10.884295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:10.884295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.884295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:10.884295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:10.884295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.884295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:10.884295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:10.884295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:10.884684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:10.884684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:10.884684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.884684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:10.884684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:10.884684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:10.884684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:10.884684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:10.884684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:10.884684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:10.884684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:10.884684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:10.884684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:10.884684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:10.884684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:10.884684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:10.884684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:10.884684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:10.884684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:10.885011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:10.885011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:10.885011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:13.759314 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:13.761581 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:13.761715 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:13.762417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:13.793808 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:13.794114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:13.794654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:13.794654 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:13.794654 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:13.794654 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:13.794654 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:13.794654 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:13.794654 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:13.794654 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:13.795264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:13.795264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:13.795264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:13.795264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:13.795264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:13.795264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:13.795264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:13.795264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:13.795264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:13.795264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:13.795264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:13.795264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:13.795538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:13.795538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:13.795538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:13.795538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:13.795538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:13.795538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:13.795538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:13.795538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:13.795538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:13.795538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:13.795538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:13.795538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:13.795538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:13.795896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:13.795896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:13.796107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:13.796107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:13.796107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:13.796107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:13.796107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:13.796107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:13.796107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:13.796107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:13.796107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:13.796107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:13.796107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:13.796107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:13.796107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:13.796107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:13.796107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:13.796107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:13.796416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:13.796416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:13.796416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:13.796416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:13.796416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:13.796416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:13.796416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:13.796416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:13.796416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:13.796416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:13.796416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:13.796416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:13.796416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:13.796416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:13.796416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:13.796416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:13.796416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:13.796416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:13.796416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:13.796788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:13.796788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:13.796788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:15.841714 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:15.842200 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:15.842200 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:15.844765 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:15.845018 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:15.845158 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:15.845291 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:15.845427 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:15.845550 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:15.845676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.846520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:15.846520 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:15.846520 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:15.846520 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:15.846520 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:15.846520 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:15.846520 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.846520 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:15.847600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.847600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:15.847600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:15.847600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:15.847600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.847600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:15.847600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:15.847600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.847600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:15.847600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:15.847600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:15.847600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.847955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:15.847955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:15.847955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:15.847955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:15.847955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:15.847955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:15.847955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:15.847955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:15.847955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:15.847955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:15.847955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:15.847955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:15.847955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:15.848368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.848368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:15.850288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.850288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:15.850288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:15.850288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:15.850288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:15.850288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:15.850288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.850288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:15.850288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:15.850288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.850288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:15.850288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:15.850288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.850288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:15.850288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:15.850288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.850872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:15.850872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:15.850872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.850872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:15.850872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:15.850872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:15.850872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.850872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:15.850872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:15.850872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:15.850872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:15.850872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:15.850872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:15.850872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:15.850872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:15.850872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:15.850872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:15.850872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:15.850872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:15.851474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:15.851474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.851474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:15.851474 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:15.852637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:15.852637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:15.852795 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:38:15.854461 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:15.855897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.857124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:15.857124 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:15.857124 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:15.857124 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:15.857124 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:15.857124 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:15.857124 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.857124 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:15.858436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.858436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:15.858436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:15.858436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:15.858436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.858436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:15.858436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:15.858436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:15.858436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.858436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:15.858436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:15.858436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:15.858436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:15.858436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:15.858436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:15.859006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:15.859006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:15.859006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:15.859006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:15.859006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:15.859006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:15.859006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:15.859006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.859006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:15.859337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.859337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:15.859337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:15.859337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:15.859337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.859337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:15.859337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:15.859337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:38:15.859337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:15.859337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:15.859337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.859337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:15.859724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:15.859724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:15.859724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.859724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:15.859724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:15.859724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:15.859724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:15.859724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:15.859724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:15.859724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:15.859724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:15.859724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:15.859724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:15.859724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:15.860117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:15.860117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:15.860117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.860117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:15.860117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.860117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:15.860117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:15.860117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:15.860117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:15.860117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:15.860117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.860117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:15.860117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:15.860117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.860613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:15.860613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:15.860613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.860613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:15.860613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:15.860613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.860613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:15.860613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:15.860613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:38:15.860613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:15.860613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:15.860613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.860613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:15.860613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:15.860613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:15.860613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.860613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:15.860613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:15.860613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:15.861181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:15.861181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:15.861181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:15.861181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:15.861181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:15.861181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:15.861181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:15.861181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:15.861181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:15.861181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:15.861181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.861181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:15.861181 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:15.861181 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:15.862665 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:15.862665 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:15.862665 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:15.862665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.862665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:15.862665 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:15.862665 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:15.862665 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:15.862665 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:15.862665 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:15.862665 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.862665 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:15.863098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.863098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:15.863098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:15.863098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:15.863098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.863098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:15.863098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:15.863098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:15.863098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.863098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:15.863098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:15.863098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:15.863098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:15.863098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:15.863098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:15.863541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:15.863541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:15.863541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:15.863541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:15.863541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:15.863541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:15.863541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:15.863541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.863541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:15.863541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.863541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:15.863541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:15.863541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:15.863541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.863884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:15.863884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:15.863884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:15.863884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:15.863884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.863884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:15.863884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:15.863884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:15.863884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.863884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:15.863884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:15.863884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:15.863884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:15.864280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:15.864280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:15.864280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:15.864280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:15.864280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:15.864280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:15.864280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:15.864280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:15.864280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:15.864280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.864280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:15.864280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.864773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:15.864773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:15.864773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:15.864773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:15.864773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:15.864773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.864773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:15.864773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:15.864773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.864773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:15.864773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:15.864773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.864773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:15.864773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:15.864773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.864773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:15.864773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:15.864773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:15.864773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.865406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:15.881496 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:15.881756 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:15.881878 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:15.882047 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:15.882363 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:15.883682 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:15.884090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.884622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:15.884622 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:15.884622 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:15.884622 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:15.884622 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:15.884622 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:15.884622 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.884622 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:15.885169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.885169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:15.885169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:15.885169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:15.885169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.885169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:15.885169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:15.885169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.885169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:15.885169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:15.885169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:15.885169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.885537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:15.885537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:15.885537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:15.885537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:15.885537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:15.885537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:15.885537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:15.885537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:15.885537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:15.885537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:15.885537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:15.885537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:15.885537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:15.885759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.885759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:15.886790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.886790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:15.886790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:15.886790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:15.886790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:15.886790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:15.886790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.886790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:15.886790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:15.886790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.886790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:15.886790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:15.886790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.886790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:15.886790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:15.886790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:15.887122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:15.887122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:15.887122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.887122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:15.887122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:15.887122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:15.887122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:15.887122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:15.887122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:15.887122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:15.887122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:15.887122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:15.887122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:15.887122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:15.887122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:15.887122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:15.887122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:15.887122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:15.887122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:15.887430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:15.887430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:15.887430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:18.763674 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:18.764335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:18.764498 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:18.765188 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:18.795289 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:18.795406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:18.795878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:18.795878 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:18.795878 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:18.795878 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:18.795878 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:18.795878 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:18.795878 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:18.795878 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:18.796469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:18.796469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:18.796469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:18.796469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:18.796469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:18.796469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:18.796469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:18.796469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:18.796469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:18.796469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:18.796469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:18.796469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:18.796793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:18.796793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:18.796793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:18.796793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:18.796793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:18.796793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:18.796793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:18.796793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:18.796793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:18.796793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:18.796793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:18.796793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:18.796793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:18.797029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:18.797029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:18.797288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:18.797288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:18.797288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:18.797288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:18.797288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:18.797288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:18.797288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:18.797288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:18.797288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:18.797288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:18.797288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:18.797288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:18.797288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:18.797288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:18.797288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:18.797288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:18.797741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:18.797741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:18.797741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:18.797741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:18.797741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:18.797741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:18.797741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:18.797741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:18.797741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:18.797741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:18.797741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:18.797741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:18.797741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:18.797741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:18.797741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:18.797741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:18.797741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:18.797741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:18.797741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:18.798263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:18.798263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:18.798263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:20.842115 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:20.842721 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:20.842781 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:20.844032 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:20.844203 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:20.844357 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:20.844465 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:20.844573 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:20.844690 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:20.844791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.845216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:20.845216 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:20.845216 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:20.845216 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:20.845216 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:20.845216 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:20.845216 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.845216 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:20.845954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.845954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:20.845954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:20.845954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:20.845954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.845954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:20.845954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:20.845954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.845954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:20.845954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:20.845954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:20.845954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.846366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:20.846366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:20.846366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:20.846366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:20.846366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:20.846366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:20.846366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:20.846366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:20.846366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:20.846366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:20.846366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:20.846366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:20.846366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:20.846917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.846917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:20.847090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.847090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:20.847090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:20.847090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:20.847090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:20.847090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:20.847090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.847090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:20.847090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:20.847090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.847090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:20.847090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:20.847090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.847090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:20.847090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:20.847090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.847476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:20.847476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:20.847476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.847476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:20.847476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:20.847476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:20.847476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.847476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:20.847476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:20.847476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:20.847476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:20.847476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:20.847476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:20.847476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:20.847476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:20.847476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:20.847476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:20.847476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:20.847476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:20.847972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:20.847972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.847972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:20.852621 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:20.853053 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:20.853198 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:20.853368 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:38:20.853769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:20.853952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.854424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:20.854424 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:20.854424 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:20.854424 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:20.854424 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:20.854424 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:20.854424 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.854424 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:20.855259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.855259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:20.855259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:20.855259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:20.855259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.855259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:20.855259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:20.855259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:20.855259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.855259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:20.855259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:20.855259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:20.855259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:20.855259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:20.855259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:20.855540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:20.855540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:20.855540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:20.855540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:20.855540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:20.855540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:20.855540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:20.855540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.855540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:20.856184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.856184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:20.856184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:20.856184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:20.856184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.856184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:20.856184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:20.856184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:38:20.856184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:20.856184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:20.856184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.856184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:20.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:20.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:20.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:20.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:20.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:20.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:20.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:20.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:20.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:20.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:20.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:20.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:20.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:20.856727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:20.856727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:20.856727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.856727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:20.857713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.857713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:20.857713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:20.857713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:20.857713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:20.857713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:20.857713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.857713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:20.857713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:20.857713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.857713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:20.857713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:20.857713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.857713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:20.857713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:20.857713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:20.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:20.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:38:20.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:20.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:20.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:20.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:20.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:20.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:20.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:20.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:20.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:20.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:20.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:20.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:20.858025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.858354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:20.860642 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:20.860849 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:20.860990 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:20.861130 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:20.861457 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:20.861700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.862151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:20.862151 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:20.862151 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:20.862151 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:20.862151 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:20.862151 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:20.862151 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.862151 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:20.862939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.862939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:20.862939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:20.862939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:20.862939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.862939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:20.862939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:20.862939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:20.862939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.862939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:20.862939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:20.862939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:20.862939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:20.862939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:20.862939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:20.863267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:20.863267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:20.863267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:20.863267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:20.863267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:20.863267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:20.863267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:20.863267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.863267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:20.863894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.863894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:20.863894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:20.863894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:20.863894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.863894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:20.863894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:20.863894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:20.863894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:20.863894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.863894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:20.864119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:20.864119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:20.864119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.864119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:20.864119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:20.864119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:20.864119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:20.864119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:20.864119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:20.864119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:20.864119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:20.864119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:20.864119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:20.864119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:20.864408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:20.864408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:20.864408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.864408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:20.865047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.865047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:20.865047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:20.865047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:20.865047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:20.865047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:20.865047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.865047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:20.865047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:20.865047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.865047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:20.865047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:20.865047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.865047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:20.865047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:20.865047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.865047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:20.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:20.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:20.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:20.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:20.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:20.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:20.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:20.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:20.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:20.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:20.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:20.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:20.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:20.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:20.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:20.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:20.865418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:20.865789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:20.865789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:20.865789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.865789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:20.881903 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:20.882294 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:20.882448 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:20.882671 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:20.882963 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:20.883163 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:20.883362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.883791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:20.883791 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:20.883791 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:20.883791 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:20.883791 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:20.883791 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:20.883791 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.883791 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:20.884388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.884388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:20.884388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:20.884388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:20.884388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.884388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:20.884388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:20.884388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.884388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:20.884388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:20.884388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:20.884388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.884655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:20.884655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:20.884655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:20.884655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:20.884655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:20.884655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:20.884655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:20.884655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:20.884655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:20.884655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:20.884655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:20.884655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:20.884655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:20.884877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.884877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:20.885029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.885029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:20.885029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:20.885029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:20.885029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:20.885029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:20.885029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.885029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:20.885029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:20.885029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.885029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:20.885029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:20.885029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.885029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:20.885029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:20.885029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:20.885439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:20.885439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:20.885439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.885439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:20.885439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:20.885439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:20.885439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:20.885439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:20.885439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:20.885439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:20.885439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:20.885439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:20.885439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:20.885439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:20.885439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:20.885439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:20.885439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:20.885439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:20.885439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:20.885838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:20.885838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:20.885838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:23.764603 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:23.765120 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:23.765120 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:23.765938 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:23.801775 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:23.801943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:23.802396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:23.802396 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:23.802396 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:23.802396 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:23.802396 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:23.802396 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:23.802396 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:23.802396 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:23.803013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:23.803013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:23.803013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:23.803013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:23.803013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:23.803013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:23.803013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:23.803013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:23.803013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:23.803013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:23.803013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:23.803013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:23.803330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:23.803330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:23.803330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:23.803330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:23.803330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:23.803330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:23.803330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:23.803330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:23.803330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:23.803330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:23.803330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:23.803330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:23.803330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:23.803670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:23.803670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:23.803739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:23.803739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:23.803739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:23.803739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:23.803739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:23.803739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:23.803739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:23.803739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:23.803739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:23.803739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:23.803739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:23.803739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:23.803739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:23.803739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:23.803739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:23.803739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:23.804066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:23.804066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:23.804066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:23.804066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:23.804066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:23.804066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:23.804066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:23.804066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:23.804066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:23.804066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:23.804066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:23.804066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:23.804066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:23.804066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:23.804066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:23.804066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:23.804066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:23.804066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:23.804066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:23.804450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:23.804450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:23.804450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:25.843278 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:25.843278 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:25.843278 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:25.844272 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:25.845437 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:25.845437 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:25.845437 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:25.845437 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:25.845437 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:25.845437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.845941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:25.845941 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:25.845941 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:25.845941 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:25.845941 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:25.845941 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:25.845941 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.845941 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:25.846573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.846573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:25.846573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:25.846573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:25.846573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.846573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:25.846573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:25.846573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.846573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:25.846573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:25.846573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:25.846573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.848672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:25.848672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:25.848672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:25.848672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:25.848672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:25.848672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:25.848672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:25.848672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:25.848672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:25.848672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:25.848672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:25.848672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:25.848672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:25.848896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.848896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:25.848896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.848896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:25.848896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:25.848896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:25.848896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:25.848896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:25.848896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.848896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:25.848896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:25.848896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.848896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:25.848896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:25.848896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.849216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:25.849216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:25.849216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.849216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:25.849216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:25.849216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.849216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:25.849216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:25.849216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:25.849216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.849216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:25.849216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:25.849216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:25.849216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:25.849216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:25.849216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:25.849216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:25.849216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:25.849216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:25.849648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:25.849648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:25.849648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:25.849648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:25.849648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.849648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:25.853109 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:25.854022 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:25.854022 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:25.854022 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:38:25.854022 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:25.854022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.854381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:25.854381 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:25.854381 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:25.854381 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:25.854381 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:25.854381 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:25.854381 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.854381 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:25.855489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.855489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:25.855489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:25.855489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:25.855489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.855489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:25.855489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:25.855489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:25.855489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.855489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:25.855489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:25.855489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:25.855489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:25.855489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:25.855489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:25.855979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:25.855979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:25.855979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:25.855979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:25.855979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:25.855979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:25.855979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:25.855979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.855979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:25.855979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.855979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:25.855979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:25.855979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:25.855979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:25.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:25.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:38:25.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:25.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:25.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:25.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:25.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:25.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:25.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:25.856465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:25.856881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:25.856881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:25.856881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:25.856881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:25.856881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:25.856881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:25.856881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:25.856881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:25.856881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:25.856881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:25.856881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.856881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:25.857239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.857239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:25.857239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:25.857239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:25.857239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:25.857239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:25.857239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.857239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:25.857239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:25.857239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.857239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:25.857239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:25.857239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.857239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:25.857239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:25.857239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.857766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:25.857766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:25.857766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:38:25.857766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:25.857766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:25.857766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.857766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:25.857766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:25.857766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:25.857766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.857766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:25.857766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:25.857766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:25.857766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:25.857766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:25.857766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:25.857766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:25.857766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:25.858236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:25.858236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:25.858236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:25.858236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:25.858236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:25.858236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.858236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:25.860933 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:25.861295 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:25.861699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:25.861699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:25.861988 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:25.862182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.863371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:25.863371 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:25.863371 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:25.863371 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:25.863371 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:25.863371 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:25.863371 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.863371 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:25.863730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.863730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:25.863730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:25.863730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:25.863730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.863730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:25.863730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:25.863730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:25.863730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.863730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:25.863730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:25.863730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:25.863730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:25.863730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:25.863730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:25.864175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:25.864175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:25.864175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:25.864175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:25.864175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:25.864175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:25.864175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:25.864175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.864175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:25.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:25.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:25.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:25.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:25.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:25.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:25.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:25.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:25.865576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:25.865576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:25.865576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.865576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:25.865576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:25.865576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:25.865576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:25.865576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:25.865576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:25.865576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:25.865576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:25.865576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:25.865576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:25.865576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:25.865973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:25.865973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:25.865973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.865973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:25.867033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.867033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:25.867033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:25.867033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:25.867033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:25.867033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:25.867033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.867033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:25.867033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:25.867033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.867033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:25.867033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:25.867033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.867033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:25.867033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:25.867033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.867033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:25.867577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:25.867577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:25.867577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:25.867577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.867577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:25.867577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:25.867577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:25.867577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.867577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:25.867577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:25.867577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:25.867577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:25.867577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:25.867577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:25.867577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:25.867577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:25.867577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:25.867577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:25.867577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:25.868075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:25.868075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:25.868075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.868075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:25.882447 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:25.883065 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:25.883065 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:25.883065 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:25.883350 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:25.883608 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:25.883785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.884333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:25.884333 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:25.884333 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:25.884333 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:25.884333 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:25.884333 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:25.884333 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.884333 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:25.885899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.885899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:25.885899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:25.885899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:25.885899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.885899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:25.885899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:25.885899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.885899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:25.885899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:25.885899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:25.885899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.886186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:25.886186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:25.886186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:25.886186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:25.886186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:25.886186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:25.886186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:25.886186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:25.886186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:25.886186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:25.886186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:25.886186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:25.886186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:25.886414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.886414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:25.886414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.886414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:25.886414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:25.886414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:25.886414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:25.886414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:25.886414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.886414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:25.886414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:25.886414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.886414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:25.886414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:25.886414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.886704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:25.886704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:25.886704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:25.886704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:25.886704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:25.886704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.886704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:25.886704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:25.886704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:25.886704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:25.886704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:25.886704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:25.886704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:25.886704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:25.886704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:25.886704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:25.886704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:25.886704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:25.886704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:25.887004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:25.887004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:25.887004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:25.887004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:25.887004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:25.887004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:28.765034 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:28.767054 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:28.767054 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:28.767054 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:28.800756 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:28.801008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:28.802509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:28.802509 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:28.802509 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:28.802509 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:28.802509 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:28.802509 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:28.802509 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:28.802509 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:28.802509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:28.802509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:28.802509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:28.802509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:28.802509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:28.802509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:28.802509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:28.802924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:28.802924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:28.802924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:28.802924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:28.802924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:28.802924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:28.802924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:28.802924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:28.802924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:28.802924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:28.802924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:28.802924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:28.802924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:28.802924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:28.803160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:28.803160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:28.803160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:28.803160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:28.803160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:28.803160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:28.804147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:28.804147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:28.804147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:28.804147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:28.804147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:28.804147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:28.804147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:28.804147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:28.804147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:28.804147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:28.804147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:28.804147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:28.804147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:28.804147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:28.804147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:28.804147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:28.804821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:28.804821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:28.804821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:28.804821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:28.804821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:28.804821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:28.804821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:28.804821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:28.804821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:28.804821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:28.804821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:28.804821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:28.804821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:28.804821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:28.804821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:28.804821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:28.804821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:28.804821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:28.804821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:28.805500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:28.805500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:28.805500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:30.843208 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:30.844480 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:30.844480 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:30.844871 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:30.845071 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:30.845208 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:30.845382 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:30.845548 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:30.845696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:30.845835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.847773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:30.847773 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:30.847773 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:30.847773 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:30.847773 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:30.847773 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:30.847773 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.847773 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:30.847773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.847773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:30.847773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:30.847773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:30.847773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.847773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:30.847773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:30.848090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.848090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:30.848090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:30.848090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:30.848090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.848090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:30.848090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:30.848090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:30.848090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:30.848090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:30.848090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:30.848090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:30.848090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:30.848090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:30.848348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:30.848348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:30.848348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:30.848348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:30.848348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.848348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:30.848348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.848348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:30.848348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:30.848348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:30.848348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:30.848348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:30.848348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.848348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:30.848633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:30.849001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:30.849001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:30.849001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:30.849001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:30.849001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:30.849001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:30.849001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:30.849001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:30.849001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.849001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:30.853512 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:30.854023 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:30.854023 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:30.854023 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:38:30.854484 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:30.854739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.855326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:30.855326 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:30.855326 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:30.855326 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:30.855326 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:30.855326 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:30.855326 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.855326 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:30.857631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.857631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:30.857631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:30.857631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:30.857631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.857631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:30.857631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:30.857631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:30.857631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.857631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:30.857631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:30.857631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:30.857631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:30.857631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:30.857631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:30.857872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:30.857872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:30.857872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:30.857872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:30.857872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:30.857872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:30.857872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:30.857872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.857872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:30.857872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.857872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:30.857872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:30.857872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:30.857872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:30.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:30.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:38:30.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:30.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:30.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:30.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:30.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:30.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:30.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:30.858111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:30.858358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:30.858358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:30.858358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:30.858358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:30.858358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:30.858358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:30.858358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:30.858358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:30.858358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:30.858358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:30.858358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.858358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:30.858669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.858669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:30.858669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:30.858669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:30.858669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:30.858669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:30.858669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.858669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:30.858669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:30.858669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.858669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:30.858669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:30.858669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.858669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:30.858669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:30.858669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.858930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:30.858930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:30.858930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:38:30.858930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:30.858930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:30.858930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.858930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:30.858930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:30.858930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:30.858930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.858930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:30.858930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:30.858930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:30.858930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:30.858930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:30.858930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:30.858930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:30.858930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:30.859326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:30.859326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:30.859326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:30.859326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:30.859326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:30.859326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.859326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:30.861367 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:30.861922 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:30.861922 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:30.861922 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:30.862484 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:30.862727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.863315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:30.863315 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:30.863315 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:30.863315 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:30.863315 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:30.863315 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:30.863315 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.863315 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:30.865794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.865794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:30.865794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:30.865794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:30.865794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.865794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:30.865794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:30.865794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:30.865794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.865794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:30.865794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:30.865794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:30.865794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:30.865794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:30.865794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:30.866025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:30.866025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:30.866025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:30.866025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:30.866025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:30.866025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:30.866025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:30.866025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.866025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:30.866025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.866025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:30.866025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:30.866025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:30.866025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.866315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:30.866315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:30.866315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:30.866315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:30.866315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.866315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:30.866315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:30.866315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:30.866315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.866315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:30.866315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:30.866315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:30.866315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:30.866537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:30.866537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:30.866537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:30.866537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:30.866537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:30.866537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:30.866537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:30.866537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:30.866537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:30.866537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.866537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:30.866537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:30.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:30.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:30.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:30.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:30.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:30.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:30.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:30.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:30.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:30.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:30.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:30.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:30.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:30.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.867054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:30.882876 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:30.883537 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:30.883537 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:30.883537 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:30.884157 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:30.884469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:30.885132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.887270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:30.887270 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:30.887270 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:30.887270 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:30.887270 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:30.887270 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:30.887270 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.887270 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:30.887270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.887270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:30.887270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:30.887270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:30.887270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.887270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:30.887270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:30.887619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.887619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:30.887619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:30.887619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:30.887619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.887619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:30.887619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:30.887619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:30.887619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:30.887619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:30.887619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:30.887619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:30.887619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:30.887619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:30.887888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:30.887888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:30.887888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:30.887888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:30.887888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.887888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:30.892275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.892275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:30.892275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:30.892275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:30.892275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:30.892275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:30.892275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.892275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:30.892275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:30.892275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.892275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:30.892275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:30.892275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.892275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:30.892275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:30.892275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:30.893578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:30.893578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:30.893578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.893578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:30.893578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:30.893578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:30.893578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:30.893578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:30.893578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:30.893578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:30.893578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:30.893578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:30.893578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:30.893578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:30.893578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:30.893578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:30.893578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:30.893578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:30.893578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:30.893907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:30.893907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:30.893907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:33.765954 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:33.766631 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:33.766631 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:33.766997 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:33.794908 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:33.794908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:33.795177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:33.795177 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:33.795177 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:33.795177 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:33.795177 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:33.795177 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:33.795177 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:33.795177 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:33.795849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:33.795849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:33.795849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:33.795849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:33.795849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:33.795849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:33.795849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:33.795849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:33.795849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:33.795849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:33.795849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:33.795849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:33.796145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:33.796145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:33.796145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:33.796145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:33.796145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:33.796145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:33.796145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:33.796145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:33.796145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:33.796145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:33.796145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:33.796145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:33.796145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:33.796506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:33.796506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:33.796674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:33.796674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:33.796674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:33.796674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:33.796674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:33.796674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:33.796674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:33.796674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:33.796674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:33.796674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:33.796674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:33.796674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:33.796674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:33.796674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:33.796674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:33.796674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:33.796984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:33.796984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:33.796984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:33.796984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:33.796984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:33.796984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:33.796984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:33.796984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:33.796984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:33.796984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:33.796984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:33.796984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:33.796984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:33.796984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:33.796984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:33.796984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:33.796984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:33.796984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:33.796984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:33.797396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:33.797396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:33.797396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:35.844068 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:35.844690 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:35.844690 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:35.845309 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:35.846321 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:35.846321 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:35.846321 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:35.846321 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:35.846321 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:35.846321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.846549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:35.846549 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:35.846549 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:35.846549 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:35.846549 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:35.846549 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:35.846549 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.846549 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:35.847083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.847083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:35.847083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:35.847083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:35.847083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.847083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:35.847083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:35.847083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.847083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:35.847083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:35.847083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:35.847083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.847359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:35.847359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:35.847359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:35.847359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:35.847359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:35.847359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:35.847359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:35.847359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:35.847359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:35.847359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:35.847359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:35.847359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:35.847359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:35.847572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.847572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:35.847862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.847862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:35.847862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:35.847862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:35.847862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:35.847862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:35.847862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.847862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:35.847862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:35.847862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.847862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:35.847862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:35.847862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.847862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:35.847862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:35.847862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.848134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:35.848134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:35.848134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.848134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:35.848134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:35.848134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:35.848134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.848134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:35.848134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:35.848134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:35.848134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:35.848134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:35.848134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:35.848134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:35.848134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:35.848134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:35.848134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:35.848134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:35.848134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:35.848478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:35.848478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.848478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:35.853857 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:35.854036 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:35.854120 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:35.854204 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:38:35.854900 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:35.854900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.855003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:35.855003 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:35.855003 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:35.855003 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:35.855003 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:35.855003 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:35.855003 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.855003 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:35.855609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.855609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:35.855609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:35.855609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:35.855609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.855609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:35.855609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:35.855609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:35.855609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.855609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:35.855609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:35.855609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:35.855609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:35.855609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:35.855609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:35.855898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:35.855898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:35.855898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:35.855898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:35.855898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:35.855898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:35.855898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:35.855898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.855898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:35.856148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.856148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:35.856148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:35.856148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:35.856148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.856148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:35.856148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:35.856148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:38:35.856148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:35.856148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:35.856148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.856148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:35.856397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:35.856397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:35.856397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.856397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:35.856397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:35.856397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:35.856397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:35.856397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:35.856397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:35.856397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:35.856397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:35.856397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:35.856397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:35.856397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:35.856682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:35.856682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:35.856682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.856682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:35.856830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.856830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:35.856830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:35.856830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:35.856830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:35.856830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:35.856830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.856830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:35.856830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:35.856830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.856830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:35.856830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:35.856830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.856830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:35.856830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:35.856830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:38:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:35.857404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:35.857404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:35.857404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:35.857404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:35.857404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:35.857404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.857404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:35.861738 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:35.861959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:35.862038 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:35.862131 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:35.862868 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:35.862868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.862966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:35.862966 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:35.862966 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:35.862966 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:35.862966 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:35.862966 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:35.862966 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.862966 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:35.863560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.863560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:35.863560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:35.863560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:35.863560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.863560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:35.863560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:35.863560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:35.863560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.863560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:35.863560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:35.863560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:35.863560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:35.863560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:35.863560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:35.863836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:35.863836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:35.863836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:35.863836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:35.863836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:35.863836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:35.863836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:35.863836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.863836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:35.864131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.864131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:35.864131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:35.864131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:35.864131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.864131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:35.864131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:35.864131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:35.864131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:35.864131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.864131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:35.864367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:35.864367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:35.864367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.864367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:35.864367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:35.864367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:35.864367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:35.864367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:35.864367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:35.864367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:35.864367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:35.864367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:35.864367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:35.864367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:35.864635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:35.864635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:35.864635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.864635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:35.864845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.864845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:35.864845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:35.864845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:35.864845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:35.864845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:35.864845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.864845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:35.864845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:35.864845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.864845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:35.864845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:35.864845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.864845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:35.864845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:35.864845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.864845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:35.865153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:35.865153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:35.865153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:35.865153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.865153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:35.865153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:35.865153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:35.865153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.865153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:35.865153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:35.865153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:35.865153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:35.865153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:35.865153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:35.865153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:35.865153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:35.865153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:35.865153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:35.865153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:35.865467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:35.865467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:35.865467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.865467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:35.883481 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:35.883696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:35.883782 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:35.883919 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:35.884907 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:35.884907 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:35.884907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.885053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:35.885053 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:35.885053 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:35.885053 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:35.885053 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:35.885053 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:35.885053 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.885053 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:35.885865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.885865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:35.885865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:35.885865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:35.885865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.885865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:35.885865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:35.885865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.885865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:35.885865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:35.885865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:35.885865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.886146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:35.886146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:35.886146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:35.886146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:35.886146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:35.886146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:35.886146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:35.886146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:35.886146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:35.886146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:35.886146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:35.886146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:35.886146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:35.886359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.886359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:35.886817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.886817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:35.886817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:35.886817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:35.886817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:35.886817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:35.886817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.886817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:35.886817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:35.886817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.886817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:35.886817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:35.886817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.886817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:35.886817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:35.886817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:35.887096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:35.887096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:35.887096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.887096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:35.887096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:35.887096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:35.887096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:35.887096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:35.887096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:35.887096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:35.887096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:35.887096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:35.887096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:35.887096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:35.887096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:35.887096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:35.887096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:35.887096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:35.887096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:35.887439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:35.887439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:35.887439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:38.766489 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:38.767072 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:38.767072 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:38.767644 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:38.798651 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:38.799637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:38.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:38.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:38.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:38.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:38.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:38.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:38.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:38.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:38.800786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:38.800786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:38.800786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:38.800786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:38.800786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:38.800786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:38.800786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:38.800786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:38.800786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:38.800786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:38.800786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:38.800786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:38.801175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:38.801175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:38.801175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:38.801175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:38.801175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:38.801175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:38.801175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:38.801175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:38.801175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:38.801175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:38.801175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:38.801175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:38.801175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:38.801513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:38.801513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:38.801871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:38.801871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:38.801871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:38.801871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:38.801871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:38.801871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:38.801871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:38.801871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:38.801871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:38.801871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:38.801871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:38.801871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:38.801871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:38.801871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:38.801871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:38.801871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:38.802134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:38.802134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:38.802134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:38.802134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:38.802134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:38.802134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:38.802134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:38.802134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:38.802134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:38.802134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:38.802134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:38.802134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:38.802134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:38.802134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:38.802134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:38.802134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:38.802134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:38.802134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:38.802134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:38.802467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:38.802467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:38.802467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:40.844527 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:40.844995 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:40.844995 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:40.846674 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:40.846674 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:40.846674 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:40.846674 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:40.846674 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:40.846816 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:40.846816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.846816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:40.846816 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:40.846816 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:40.846816 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:40.846816 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:40.846816 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:40.846816 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.846816 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:40.847382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.847382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:40.847382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:40.847382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:40.847382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.847382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:40.847382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:40.847382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.847382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:40.847382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:40.847382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:40.847382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.847649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:40.847649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:40.847649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:40.847649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:40.847649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:40.847649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:40.847649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:40.847649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:40.847649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:40.847649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:40.847649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:40.847649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:40.847649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:40.847869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.847869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:40.848136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.848136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:40.848136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:40.848136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:40.848136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:40.848136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:40.848136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.848136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:40.848136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:40.848136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.848136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:40.848136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:40.848136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.848136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:40.848136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:40.848136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.848414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:40.848414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:40.848414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.848414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:40.848414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:40.848414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:40.848414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.848414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:40.848414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:40.848414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:40.848414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:40.848414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:40.848414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:40.848414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:40.848414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:40.848414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:40.848414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:40.848414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:40.848414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:40.848779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:40.848779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.848779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:40.854095 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:40.854316 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:40.854404 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:40.854490 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:38:40.855099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:40.855099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.855187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:40.855187 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:40.855187 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:40.855187 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:40.855187 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:40.855187 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:40.855187 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.855187 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:40.855718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.855718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:40.855718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:40.855718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:40.855718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.855718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:40.855718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:40.855718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:40.855718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.855718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:40.855718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:40.855718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:40.855718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:40.855718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:40.855718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:40.855982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:40.855982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:40.855982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:40.855982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:40.855982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:40.855982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:40.855982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:40.855982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.855982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:40.856344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.856344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:40.856344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:40.856344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:40.856344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.856344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:40.856344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:40.856344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:38:40.856344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:40.856344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:40.856344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.856344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:40.856566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:40.856566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:40.856566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.856566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:40.856566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:40.856566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:40.856566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:40.856566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:40.856566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:40.856566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:40.856566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:40.856566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:40.856566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:40.856566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:40.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:40.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:40.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.856785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:40.857032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.857032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:40.857032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:40.857032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:40.857032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:40.857032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:40.857032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.857032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:40.857032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:40.857032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.857032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:40.857032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:40.857032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.857032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:40.857032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:40.857032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.857308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:40.857308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:40.857308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:38:40.857308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:40.857308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:40.857308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.857308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:40.857308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:40.857308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:40.857308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.857308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:40.857308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:40.857308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:40.857308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:40.857308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:40.857308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:40.857308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:40.857308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:40.857589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:40.857589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:40.857589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:40.857589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:40.857589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:40.857589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.857589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:40.861983 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:40.862161 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:40.862238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:40.862351 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:40.862961 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:40.862961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.863041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:40.863041 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:40.863041 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:40.863041 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:40.863041 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:40.863041 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:40.863041 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.863041 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:40.863540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.863540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:40.863540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:40.863540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:40.863540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.863540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:40.863540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:40.863540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:40.863540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.863540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:40.863540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:40.863540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:40.863540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:40.863540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:40.863540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:40.863848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:40.863848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:40.863848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:40.863848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:40.863848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:40.863848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:40.863848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:40.863848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.863848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:40.864180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.864180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:40.864180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:40.864180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:40.864180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.864180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:40.864180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:40.864180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:40.864180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:40.864180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.864180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:40.864405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:40.864405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:40.864405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.864405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:40.864405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:40.864405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:40.864405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:40.864405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:40.864405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:40.864405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:40.864405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:40.864405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:40.864405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:40.864405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:40.864629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:40.864629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:40.864629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.864629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:40.864894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.864894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:40.864894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:40.864894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:40.864894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:40.864894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:40.864894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.864894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:40.864894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:40.864894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.864894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:40.864894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:40.864894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.864894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:40.864894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:40.864894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.864894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:40.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:40.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:40.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:40.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:40.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:40.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:40.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:40.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:40.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:40.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:40.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:40.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:40.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:40.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:40.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:40.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:40.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:40.865475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:40.865475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:40.865475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.865475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:40.883760 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:40.884010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:40.884100 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:40.884223 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:40.885140 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:40.885140 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:40.885140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.885244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:40.885244 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:40.885244 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:40.885244 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:40.885244 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:40.885244 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:40.885244 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.885244 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:40.885877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.885877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:40.885877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:40.885877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:40.885877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.885877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:40.885877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:40.885877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.885877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:40.885877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:40.885877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:40.885877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.886096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:40.886096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:40.886096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:40.886096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:40.886096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:40.886096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:40.886096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:40.886096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:40.886096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:40.886096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:40.886096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:40.886096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:40.886096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:40.886350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.886350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:40.886619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.886619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:40.886619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:40.886619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:40.886619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:40.886619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:40.886619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.886619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:40.886619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:40.886619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.886619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:40.886619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:40.886619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.886619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:40.886619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:40.886619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:40.886884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:40.886884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:40.886884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.886884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:40.886884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:40.886884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:40.886884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:40.886884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:40.886884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:40.886884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:40.886884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:40.886884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:40.886884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:40.886884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:40.886884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:40.886884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:40.886884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:40.886884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:40.886884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:40.887175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:40.887175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:40.887175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:43.766852 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:43.767415 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:43.767415 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:43.768225 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:43.794964 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:43.795080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:43.795510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:43.795510 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:43.795510 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:43.795510 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:43.795510 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:43.795510 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:43.795510 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:43.795510 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:43.796320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:43.796320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:43.796320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:43.796320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:43.796320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:43.796320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:43.796320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:43.796320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:43.796320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:43.796320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:43.796320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:43.796320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:43.796754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:43.796754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:43.796754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:43.796754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:43.796754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:43.796754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:43.796754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:43.796754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:43.796754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:43.796754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:43.796754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:43.796754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:43.796754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:43.797141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:43.797141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:43.797497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:43.797497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:43.797497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:43.797497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:43.797497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:43.797497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:43.797497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:43.797497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:43.797497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:43.797497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:43.797497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:43.797497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:43.797497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:43.797497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:43.797497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:43.797497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:43.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:43.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:43.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:43.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:43.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:43.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:43.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:43.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:43.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:43.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:43.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:43.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:43.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:43.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:43.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:43.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:43.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:43.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:43.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:43.798955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:43.798955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:43.798955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:45.845087 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:45.845625 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:45.845625 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:45.846628 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:45.846773 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2cbd0c22-7c21-4b6d-94ed-2faf56386bff, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:45.846888 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:45.846992 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:45.847088 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:45.847185 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a25665e1-227a-4949-8ee2-7bc2da5f3337, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:45.847307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.847751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:45.847751 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:45.847751 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:45.847751 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:45.847751 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:45.847751 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:45.847751 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.847751 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:45.848782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.848782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:45.848782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:45.848782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:45.848782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.848782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:45.848782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:45.848782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.848782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:45.848782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:45.848782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:45.848782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.849018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:45.849018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:45.849018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:45.849018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:45.849018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:45.849018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:45.849018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:45.849018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:45.849018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:45.849018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:45.849018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:45.849018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:45.849018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:45.849220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.849220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:45.850009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.850009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:45.850009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:45.850009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:45.850009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:45.850009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:45.850009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.850009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:45.850009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:45.850009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.850009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:45.850009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:45.850009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.850009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:45.850009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:45.850009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.850343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:45.850343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:45.850343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.850343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:45.850343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:45.850343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:45.850343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.850343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:45.850343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:45.850343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:45.850343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:45.850343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:45.850343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:45.850343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:45.850343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:45.850343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:45.850343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:45.850343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:45.850343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:45.850714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:45.850714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.850714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:45.854412 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:45.854909 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:45.855035 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:45.855137 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:38:45.855500 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:45.855705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.856224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:45.856224 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:45.856224 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:45.856224 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:45.856224 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:45.856224 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:45.856224 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.856224 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:45.857046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.857046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:45.857046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:45.857046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:45.857046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.857046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:45.857046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:45.857046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:45.857046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.857046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:45.857046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:45.857046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:45.857046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:45.857046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:45.857046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:45.857554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:45.857554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:45.857554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:45.857554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:45.857554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:45.857554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:45.857554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:45.857554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.857554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:45.857851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.857851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:45.857851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:45.857851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:45.857851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.857851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:45.857851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:45.857851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:38:45.857851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:45.857851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:45.857851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.857851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:45.858222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:45.858222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:45.858222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.858222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:45.858222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:45.858222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:45.858222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:45.858222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:45.858222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:45.858222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:45.858222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:45.858222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:45.858222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:45.858222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:45.858663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:45.858663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:45.858663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.858663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:45.858663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.858663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:45.858663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:45.858663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:45.858663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:45.858663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:45.858663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.858663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:45.858663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:45.858663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.859049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:45.859049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:45.859049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.859049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:45.859049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:45.859049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.859049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:45.859049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:45.859049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:38:45.859049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:45.859049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:45.859049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.859049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:45.859049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:45.859049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:45.859049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.859049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:45.859049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:45.859049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:45.859563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:45.859563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:45.859563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:45.859563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:45.859563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:45.859563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:45.859563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:45.859563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:45.859563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:45.859563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:45.859563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.859563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:45.862377 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:45.862679 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:45.862781 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:45.862884 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:45.863196 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:45.863433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.863881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:45.863881 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:45.863881 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:45.863881 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:45.863881 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:45.863881 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:45.863881 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.863881 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:45.864469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.864469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:45.864469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:45.864469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:45.864469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.864469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:45.864469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:45.864469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:45.864469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.864469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:45.864469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:45.864469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:45.864469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:45.864469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:45.864469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:45.864933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:45.864933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:45.864933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:45.864933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:45.864933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:45.864933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:45.864933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:45.864933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.864933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:45.865184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.865184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:45.865184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:45.865184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:45.865184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.865184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:45.865184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:45.865184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:45.865184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:45.865184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.865184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:45.865530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:45.865530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:45.865530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.865530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:45.865530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:45.865530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:45.865530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:45.865530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:45.865530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:45.865530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:45.865530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:45.865530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:45.865530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:45.865530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:45.865928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:45.865928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:45.865928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.865928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:45.865928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.865928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:45.865928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:45.865928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:45.865928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:45.865928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:45.865928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.865928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:45.865928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:45.865928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.866408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:45.866408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:45.866408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.866408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:45.866408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:45.866408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.866408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:45.866408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:45.866408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:45.866408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:45.866408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.866408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:45.866408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:45.866408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:45.866408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.866408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:45.866408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:45.866408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:45.866408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:45.866893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:45.866893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:45.866893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:45.866893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:45.866893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:45.866893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:45.866893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:45.866893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:45.866893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:45.866893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.866893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:45.884162 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:45.884692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:45.884873 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:45.885096 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:45.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:45.885747 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:45.886004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.886561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:45.886561 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:45.886561 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:45.886561 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:45.886561 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:45.886561 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:45.886561 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.886561 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:45.887665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.887665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:45.887665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:45.887665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:45.887665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.887665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:45.887665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:45.887665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.887665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:45.887665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:45.887665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:45.887665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.888047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:45.888047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:45.888047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:45.888047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:45.888047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:45.888047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:45.888047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:45.888047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:45.888047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:45.888047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:45.888047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:45.888047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:45.888047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:45.888421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.888421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:45.888864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.888864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:45.888864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:45.888864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:45.888864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:45.888864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:45.888864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.888864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:45.888864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:45.888864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.888864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:45.888864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:45.888864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.888864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:45.888864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:45.888864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:45.889369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:45.889369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:45.889369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.889369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:45.889369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:45.889369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:45.889369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:45.889369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:45.889369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:45.889369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:45.889369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:45.889369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:45.889369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:45.889369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:45.889369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:45.889369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:45.889369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:45.889369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:45.889369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:45.889910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:45.889910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:45.889910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:48.767271 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:48.767841 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:48.767892 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:48.768804 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:48.809277 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:48.809512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:48.810078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:48.810078 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:48.810078 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:48.810078 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:48.810078 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:48.810078 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:48.810078 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:48.810078 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:48.810831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:48.810831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:48.810831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:48.810831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:48.810831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:48.810831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:48.810831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:48.810831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:48.810831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:48.810831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:48.810831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:48.810831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:48.811076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:48.811076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:48.811076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:48.811076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:48.811076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:48.811076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:48.811076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:48.811076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:48.811076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:48.811076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:48.811076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:48.811076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:48.811076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:48.811334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:48.811334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:48.811577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:48.811577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:48.811577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:48.811577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:48.811577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:48.811577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:48.811577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:48.811577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:48.811577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:48.811577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:48.811577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:48.811577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:48.811577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:48.811577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:48.811577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:48.811577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:48.811900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:48.811900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:48.811900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:48.811900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:48.811900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:48.811900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:48.811900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:48.811900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:48.811900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:48.811900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:48.811900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:48.811900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:48.811900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:48.811900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:48.811900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:48.811900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:48.811900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:48.811900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:48.811900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:48.812218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:48.812218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:48.812218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:50.845644 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:50.846044 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:50.846078 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:50.846768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:50.846962 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:50.847087 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:50.847217 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:50.847352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.849102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:50.849102 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:50.849102 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:50.849102 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:50.849102 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:50.849102 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:50.849102 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.849102 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:50.849853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.849853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:50.849853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:50.849853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:50.849853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.849853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:50.849853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:50.849853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.849853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:50.849853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:50.849853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:50.849853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.850179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:50.850179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:50.850179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:50.850179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:50.850179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:50.850179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:50.850179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:50.850179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:50.850179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:50.850179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:50.850179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:50.850179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:50.850179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:50.850432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.850432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:50.857422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.857422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:50.857422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:50.857422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:50.857422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:50.857422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:50.857422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.857422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:50.857422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:50.857422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.857422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:50.857422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:50.857422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.857422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:50.857422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:50.857422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.857765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:50.857765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:50.857765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.857765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:50.857765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:50.857765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:50.857765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.857765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:50.857765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:50.857765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:50.857765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:50.857765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:50.857765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:50.857765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:50.857765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:50.857765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:50.857765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:50.857765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:50.857765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:50.858338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:50.858338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.858338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:50.858338 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:50.860243 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:50.861204 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:50.861331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:38:50.862732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:50.865744 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:50.865744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.865744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:50.865744 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:50.865744 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:50.865744 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:50.865744 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:50.865744 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:50.865744 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.865744 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:50.866551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.866551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:50.866551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:50.866551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:50.866551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.866551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:50.866551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:50.866551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:50.866551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.866551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:50.866551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:50.866551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:50.866551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:50.866551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:50.866551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:50.866874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:50.866874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:50.866874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:50.866874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:50.866874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:50.866874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:50.866874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:50.866874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.866874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:50.867071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.867071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:50.867071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:50.867071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:50.867071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.867071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:50.867071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:50.867071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:38:50.867071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:50.867071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:50.867071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.867071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:50.867335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:50.867335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:50.867335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.867335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:50.867335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:50.867335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:50.867335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:50.867335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:50.867335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:50.867335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:50.867335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:50.867335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:50.867335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:50.867335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:50.867585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:50.867585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:50.867585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.867585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:50.867803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.867803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:50.867803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:50.867803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:50.867803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:50.867803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:50.867803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.867803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:50.867803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:50.867803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.867803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:50.867803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:50.867803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.867803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:50.867803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:50.867803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.868226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:50.868226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:50.868226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:38:50.868226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:50.868226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:50.868226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.868226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:50.868226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:50.868226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:50.868226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.868226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:50.868226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:50.868226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:50.868226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:50.868226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:50.868226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:50.868226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:50.868226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:50.868632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:50.868632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:50.868632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:50.868632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:50.868632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:50.868632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.868632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:50.868632 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:50.868632 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:50.868632 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:50.868819 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:50.868819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.869064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:50.869064 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:50.869064 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:50.869064 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:50.869064 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:50.869064 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:50.869064 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.869064 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:50.869550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.869550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:50.869550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:50.869550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:50.869550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.869550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:50.869550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:50.869550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:50.869550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.869550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:50.869550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:50.869550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:50.869550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:50.869550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:50.869550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:50.869820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:50.869820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:50.869820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:50.869820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:50.869820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:50.869820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:50.869820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:50.869820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.869820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:50.870112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.870112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:50.870112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:50.870112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:50.870112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.870112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:50.870112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:50.870112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:50.870112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:50.870112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.870112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:50.870369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:50.870369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:50.870369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.870369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:50.870369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:50.870369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:50.870369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:50.870369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:50.870369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:50.870369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:50.870369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:50.870369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:50.870369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:50.870369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:50.870618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:50.870618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:50.870618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.870618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:50.870862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.870862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:50.870862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:50.870862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:50.870862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:50.870862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:50.870862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.870862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:50.870862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:50.870862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.870862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:50.870862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:50.870862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.870862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:50.870862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:50.870862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.870862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:50.871149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:50.871149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:50.871149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:50.871149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.871149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:50.871149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:50.871149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:50.871149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.871149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:50.871149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:50.871149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:50.871149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:50.871149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:50.871149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:50.871149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:50.871149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:50.871149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:50.871149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:50.871149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:50.871480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:50.871480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:50.871480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.871480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:50.884768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:50.884942 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:50.885015 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:50.885150 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:50.885494 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:50.885714 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:50.885863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.886264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:50.886264 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:50.886264 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:50.886264 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:50.886264 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:50.886264 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:50.886264 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.886264 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:50.886834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.886834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:50.886834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:50.886834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:50.886834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.886834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:50.886834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:50.886834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.886834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:50.886834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:50.886834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:50.886834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.887059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:50.887059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:50.887059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:50.887059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:50.887059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:50.887059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:50.887059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:50.887059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:50.887059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:50.887059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:50.887059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:50.887059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:50.887059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:50.887416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.887416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:50.887491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.887491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:50.887491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:50.887491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:50.887491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:50.887491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:50.887491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.887491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:50.887491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:50.887491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.887491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:50.887491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:50.887491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.887491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:50.887491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:50.887491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:50.887769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:50.887769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:50.887769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.887769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:50.887769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:50.887769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:50.887769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:50.887769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:50.887769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:50.887769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:50.887769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:50.887769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:50.887769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:50.887769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:50.887769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:50.887769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:50.887769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:50.887769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:50.887769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:50.888083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:50.888083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:50.888083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:53.767720 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:53.768297 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:53.768297 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:53.768607 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:53.775540 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=240 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:38:53.797462 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:53.797664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:53.798166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:53.798166 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:53.798166 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:53.798166 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:53.798166 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:53.798166 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:53.798166 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:53.798166 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:53.798837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:53.798837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:53.798837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:53.798837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:53.798837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:53.798837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:53.798837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:53.798837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:53.798837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:53.798837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:53.798837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:53.798837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:53.799114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:53.799114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:53.799114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:53.799114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:53.799114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:53.799114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:53.799114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:53.799114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:53.799114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:53.799114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:53.799114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:53.799114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:53.799114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:53.799373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:53.799373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:53.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:53.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:53.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:53.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:53.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:53.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:53.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:53.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:53.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:53.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:53.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:53.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:53.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:53.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:53.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:53.799769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:53.800155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:53.800155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:53.800155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:53.800155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:53.800155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:53.800155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:53.800155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:53.800155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:53.800155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:53.800155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:53.800155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:53.800155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:53.800155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:53.800155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:53.800155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:53.800155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:53.800155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:53.800155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:53.800155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:53.800580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:53.800580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:53.800580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:55.846687 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:55.847157 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:55.847157 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:55.847844 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:55.848106 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:55.848263 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:55.848387 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:55.848494 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:55.848594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.849068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:55.849068 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:55.849068 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:55.849068 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:55.849068 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:55.849068 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:55.849068 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.849068 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:55.849841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.849841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:55.849841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:55.849841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:55.849841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.849841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:55.849841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:55.849841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.849841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:55.849841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:55.849841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:55.849841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.850153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:55.850153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:55.850153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:55.850153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:55.850153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:55.850153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:55.850153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:55.850153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:55.850153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:55.850153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:55.850153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:55.850153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:55.850153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:55.850410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.850410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:55.850830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.850830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:55.850830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:55.850830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:55.850830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:55.850830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:55.850830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.850830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:55.850830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:55.850830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.850830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:55.850830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:55.850830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.850830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:55.850830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:55.850830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.851123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:38:55.851123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:55.851123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.851123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:55.851123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:55.851123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:55.851123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.851123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:55.851123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:55.851123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:55.851123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:55.851123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:55.851123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:55.851123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:55.851123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:55.851123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:55.851123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:55.851123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:55.851123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:55.851445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:55.851445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.851445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:55.860378 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:55.860771 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:55.860904 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:55.861048 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:38:55.861456 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:55.861640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.862169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:55.862169 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:55.862169 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:55.862169 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:55.862169 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:55.862169 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:55.862169 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.862169 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:55.863052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.863052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:55.863052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:55.863052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:55.863052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.863052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:55.863052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:55.863052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:55.863052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.863052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:55.863052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:55.863052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:55.863052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:55.863052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:55.863052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:55.863352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:55.863352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:55.863352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:55.863352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:55.863352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:55.863352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:55.863352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:55.863352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.863352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:55.863990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.863990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:55.863990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:55.863990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:55.863990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.863990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:55.863990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:55.863990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:38:55.863990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:55.863990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:55.863990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.863990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:55.864231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:55.864231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:55.864231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.864231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:55.864231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:55.864231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:55.864231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:55.864231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:55.864231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:55.864231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:55.864231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:55.864231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:55.864231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:55.864231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:55.864475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:55.864475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:55.864475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.864475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:55.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:55.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:55.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:55.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:55.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:55.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:55.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:55.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:55.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:55.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:55.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:55.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.865233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:38:55.865233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:38:55.865233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:38:55.865233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:55.865233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:55.865233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.865233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:55.865233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:55.865233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:55.865233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.865233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:55.865233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:55.865233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:55.865233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:55.865233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:55.865233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:55.865233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:55.865233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:55.865559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:55.865559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:55.865559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:55.865559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:55.865559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:55.865559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.865559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:55.868151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:55.868441 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:55.868552 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:55.868699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:55.869095 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:55.869323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.869790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:55.869790 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:55.869790 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:55.869790 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:55.869790 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:55.869790 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:55.869790 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.869790 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:55.870403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.870403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:38:55.870403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:55.870403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:55.870403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.870403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:55.870403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:55.870403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:38:55.870403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.870403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:38:55.870403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:55.870403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:38:55.870403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:55.870403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:38:55.870403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:38:55.870676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:55.870676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:38:55.870676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:55.870676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:38:55.870676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:38:55.870676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:55.870676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:38:55.870676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.870676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:38:55.871322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.871322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:55.871322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:55.871322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:55.871322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.871322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:55.871322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:55.871322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:55.871322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:55.871322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.871322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:55.871552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:55.871552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:55.871552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.871552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:55.871552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:55.871552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:55.871552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:55.871552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:55.871552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:55.871552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:55.871552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:55.871552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:55.871552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:55.871552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:55.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:55.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:55.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:55.872115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.872115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:55.872115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:55.872115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:55.872115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:55.872115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:55.872115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.872115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:55.872115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:55.872115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.872115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:55.872115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:55.872115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.872115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:55.872115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:55.872115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.872115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:38:55.872455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:38:55.872455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:38:55.872455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:38:55.872455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.872455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:55.872455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:55.872455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:55.872455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.872455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:55.872455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:55.872455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:55.872455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:55.872455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:55.872455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:55.872455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:55.872455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:55.872455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:55.872455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:55.872455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:55.872767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:55.872767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:55.872767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.872767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:55.885088 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:55.885399 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:55.885496 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:55.885650 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:55.885957 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:55.886170 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:55.886360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.886810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:55.886810 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:55.886810 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:55.886810 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:55.886810 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:55.886810 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:55.886810 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.886810 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:55.887429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.887429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:55.887429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:55.887429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:55.887429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.887429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:55.887429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:55.887429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.887429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:55.887429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:55.887429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:55.887429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.887684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:55.887684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:55.887684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:55.887684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:55.887684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:55.887684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:55.887684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:55.887684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:55.887684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:55.887684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:55.887684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:55.887684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:55.887684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:55.887901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.887901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:55.888080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.888080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:55.888080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:55.888080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:55.888080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:55.888080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:55.888080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.888080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:55.888080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:55.888080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.888080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:55.888080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:55.888080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.888080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:55.888080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:55.888080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:55.888390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:38:55.888390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:55.888390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.888390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:55.888390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:55.888390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:55.888390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:55.888390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:55.888390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:55.888390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:55.888390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:55.888390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:55.888390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:55.888390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:55.888390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:55.888390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:55.888390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:55.888390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:55.888390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:55.888746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:55.888746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:55.888746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:38:58.768220 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:38:58.768704 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:38:58.768704 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:38:58.769181 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:38:58.808266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:38:58.808471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:58.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:38:58.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:58.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:38:58.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:38:58.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:58.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:38:58.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:58.808881 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:38:58.809513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:58.809513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:38:58.809513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:58.809513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:38:58.809513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:58.809513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:58.809513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:58.809513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:58.809513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:58.809513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:58.809513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:38:58.809513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:58.809764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:38:58.809764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:58.809764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:38:58.809764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:58.809764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:38:58.809764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:38:58.809764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:58.809764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:38:58.809764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:58.809764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:38:58.809764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:38:58.809764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:58.809764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:38:58.809992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:58.809992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:38:58.810220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:58.810220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:38:58.810220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:38:58.810220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:38:58.810220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:38:58.810220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:38:58.810220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:58.810220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:38:58.810220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:38:58.810220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:58.810220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:38:58.810220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:38:58.810220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:38:58.810220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:38:58.810220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:38:58.810220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:38:58.810577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:38:58.810577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:38:58.810577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:58.810577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:58.810577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:38:58.810577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:38:58.810577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:38:58.810577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:38:58.810577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:38:58.810577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:38:58.810577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:38:58.810577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:38:58.810577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:38:58.810577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:38:58.810577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:38:58.810577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:38:58.810577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:38:58.810577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:38:58.810577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:38:58.810885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:38:58.810885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:38:58.810885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:00.847551 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:00.848946 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:00.849044 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:00.850227 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:00.850227 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:00.850349 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:00.850453 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:00.850571 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:00.850818 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:00.850937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.851547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:00.851547 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:00.851547 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:00.851547 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:00.851547 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:00.851547 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:00.851547 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.851547 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:00.852234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.852234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:00.852234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:00.852234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:00.852234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.852234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:00.852234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:00.852234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.852234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:00.852234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:00.852234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:00.852234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.852512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:00.852512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:00.852512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:00.852512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:00.852512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:00.852512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:00.852512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:00.852512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:00.852512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:00.852512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:00.852512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:00.852512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:00.852512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:00.852818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.852818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:00.853126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.853126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:00.853126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:00.853126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:00.853126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:00.853126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:00.853126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.853126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:00.853126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:00.853126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.853126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:00.853126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:00.853126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.853126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:00.853126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:00.853126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.853455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:00.853455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:00.853455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.853455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:00.853455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:00.853455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:00.853455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.853455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:00.853455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:00.853455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:00.853455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:00.853455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:00.853455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:00.853455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:00.853455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:00.853455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:00.853455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:00.853455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:00.853455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:00.853770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:00.853770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.853770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:00.860752 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:00.861013 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:00.861099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:00.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:39:00.861637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:00.861812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.862263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:00.862263 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:00.862263 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:00.862263 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:00.862263 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:00.862263 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:00.862263 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.862263 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:00.862868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.862868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:00.862868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:00.862868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:00.862868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.862868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:00.862868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:00.862868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:00.862868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.862868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:00.862868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:00.862868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:00.862868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:00.862868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:00.862868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:00.863172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:00.863172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:00.863172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:00.863172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:00.863172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:00.863172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:00.863172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:00.863172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.863172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:00.863466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.863466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:00.863466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:00.863466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:00.863466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.863466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:00.863466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:00.863466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:39:00.863466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:00.863466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:00.863466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.863466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:00.863716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:00.863716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:00.863716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.863716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:00.863716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:00.863716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:00.863716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:00.863716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:00.863716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:00.863716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:00.863716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:00.863716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:00.863716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:00.863716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:00.863943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:00.863943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:00.863943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.863943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:00.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:00.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:00.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:00.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:00.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:00.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:00.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:00.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:00.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:00.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:00.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:00.864777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:00.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:00.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:39:00.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:00.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:00.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:00.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:00.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:00.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:00.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:00.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:00.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:00.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:00.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:00.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:00.865171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:00.865504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:00.865504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:00.865504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:00.865504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:00.865504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:00.865504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.865504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:00.868462 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:00.868701 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:00.868811 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:00.868940 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:00.869308 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:00.869513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.869934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:00.869934 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:00.869934 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:00.869934 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:00.869934 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:00.869934 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:00.869934 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.869934 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:00.870622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.870622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:00.870622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:00.870622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:00.870622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.870622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:00.870622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:00.870622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:00.870622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.870622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:00.870622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:00.870622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:00.870622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:00.870622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:00.870622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:00.870907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:00.870907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:00.870907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:00.870907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:00.870907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:00.870907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:00.870907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:00.870907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.870907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:00.871447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.871447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:00.871447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:00.871447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:00.871447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.871447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:00.871447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:00.871447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:00.871447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:00.871447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.871447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:00.871668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:00.871668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:00.871668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.871668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:00.871668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:00.871668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:00.871668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:00.871668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:00.871668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:00.871668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:00.871668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:00.871668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:00.871668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:00.871668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:00.872004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:00.872004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:00.872004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.872004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:00.873729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.873729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:00.873729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:00.873729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:00.873729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:00.873729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:00.873729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.873729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:00.873729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:00.873729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.873729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:00.873729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:00.873729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.873729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:00.873729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:00.873729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.873729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:00.874024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:00.874024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:00.874024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:00.874024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.874024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:00.874024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:00.874024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:00.874024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.874024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:00.874024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:00.874024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:00.874024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:00.874024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:00.874024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:00.874024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:00.874024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:00.874024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:00.874024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:00.874024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:00.874384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:00.874384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:00.874384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.874384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:00.885465 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:00.885799 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:00.885881 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:00.886015 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:00.886443 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:00.886696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:00.886912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.887430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:00.887430 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:00.887430 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:00.887430 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:00.887430 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:00.887430 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:00.887430 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.887430 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:00.888157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.888157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:00.888157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:00.888157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:00.888157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.888157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:00.888157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:00.888157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.888157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:00.888157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:00.888157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:00.888157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.888409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:00.888409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:00.888409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:00.888409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:00.888409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:00.888409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:00.888409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:00.888409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:00.888409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:00.888409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:00.888409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:00.888409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:00.888409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:00.888770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.888770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:00.889067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.889067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:00.889067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:00.889067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:00.889067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:00.889067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:00.889067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.889067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:00.889067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:00.889067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.889067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:00.889067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:00.889067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.889067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:00.889067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:00.889067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:00.889496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:00.889496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:00.889496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.889496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:00.889496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:00.889496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:00.889496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:00.889496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:00.889496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:00.889496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:00.889496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:00.889496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:00.889496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:00.889496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:00.889496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:00.889496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:00.889496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:00.889496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:00.889496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:00.889907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:00.889907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:00.889907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:03.768639 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:03.769335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:03.769553 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:03.770549 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:03.812455 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:03.812645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:03.813360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:03.813360 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:03.813360 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:03.813360 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:03.813360 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:03.813360 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:03.813360 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:03.813360 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:03.814193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:03.814193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:03.814193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:03.814193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:03.814193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:03.814193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:03.814193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:03.814193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:03.814193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:03.814193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:03.814193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:03.814193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:03.814755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:03.814755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:03.814755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:03.814755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:03.814755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:03.814755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:03.814755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:03.814755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:03.814755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:03.814755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:03.814755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:03.814755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:03.814755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:03.815215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:03.815215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:03.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:03.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:03.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:03.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:03.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:03.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:03.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:03.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:03.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:03.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:03.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:03.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:03.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:03.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:03.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:03.815755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:03.816689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:03.816689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:03.816689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:03.816689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:03.816689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:03.816689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:03.816689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:03.816689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:03.816689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:03.816689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:03.816689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:03.816689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:03.816689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:03.816689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:03.816689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:03.816689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:03.816689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:03.816689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:03.816689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:03.817365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:03.817365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:03.817365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:05.848888 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:05.849570 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:05.849570 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:05.849887 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:05.850020 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fac6a3e7-3e11-430c-adbc-035f7b9c9fd4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:05.850146 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:05.850293 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-82dfba57-356c-4cfd-80a4-83ff49127866, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:05.850400 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:05.850507 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:05.850648 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.851036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:05.851036 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:05.851036 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:05.851036 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:05.851036 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:05.851036 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:05.851036 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.851036 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:05.851725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.851725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:05.851725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:05.851725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:05.851725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.851725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:05.851725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:05.851725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.851725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:05.851725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:05.851725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:05.851725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.852119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:05.852119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:05.852119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:05.852119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:05.852119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:05.852119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:05.852119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:05.852119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:05.852119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:05.852119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:05.852119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:05.852119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:05.852119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:05.852509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.852509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:05.852509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.852509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:05.852509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:05.852509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:05.852509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:05.852509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:05.852509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.852509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:05.852509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:05.852509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.852509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:05.852509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:05.852509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.852951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:05.852951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:05.852951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.852951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:05.852951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:05.852951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.852951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:05.852951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:05.852951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:05.852951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.852951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:05.852951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:05.852951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:05.852951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:05.852951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:05.852951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:05.852951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:05.852951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:05.852951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:05.853480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:05.853480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:05.853480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:05.853480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:05.853480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.853480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:05.861186 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:05.862286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:05.862286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:05.862286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:39:05.862286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:05.862496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.862842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:05.862842 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:05.862842 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:05.862842 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:05.862842 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:05.862842 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:05.862842 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.862842 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:05.863622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.863622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:05.863622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:05.863622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:05.863622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.863622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:05.863622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:05.863622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:05.863622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.863622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:05.863622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:05.863622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:05.863622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:05.863622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:05.863622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:05.864190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:05.864190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:05.864190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:05.864190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:05.864190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:05.864190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:05.864190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:05.864190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.864190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:05.864958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.864958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:05.864958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:05.864958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:05.864958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.864958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:05.864958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:05.864958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:39:05.864958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:05.864958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:05.864958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.864958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:05.865320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:05.865320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:05.865320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.865320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:05.865320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:05.865320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:05.865320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:05.865320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:05.865320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:05.865320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:05.865320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:05.865320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:05.865320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:05.865320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:05.865698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:05.865698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:05.865698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.865698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:05.865698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.865698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:05.865698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:05.865698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:05.865698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:05.865698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:05.865698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.865698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:05.865698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:05.865698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.866081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:05.866081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:05.866081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.866081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:05.866081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:05.866081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.866081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:05.866081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:05.866081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:39:05.866081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:05.866081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:05.866081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.866081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:05.866081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:05.866081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:05.866081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.866081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:05.866081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:05.866081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:05.866684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:05.866684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:05.866684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:05.866684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:05.866684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:05.866684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:05.866684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:05.866684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:05.866684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:05.866684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:05.866684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.866684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:05.871541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:05.871541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:05.871541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:05.871541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:05.871541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:05.871541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:05.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:05.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:05.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:05.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:05.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:05.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:05.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:05.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:05.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:05.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:05.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:05.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:05.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.871786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:05.872330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:05.872330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:05.872330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:05.872330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:05.872330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:05.872330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:05.872330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:05.872330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:05.872330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:05.872330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:05.872330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:05.872330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:05.872330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.872330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:05.872330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.872833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:05.872833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:05.872833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:05.872833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.872833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:05.872833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:05.872833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:05.872833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:05.872833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.872833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:05.872833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:05.872833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:05.872833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.873205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:05.873205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:05.873205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:05.873205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:05.873205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:05.873205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:05.873205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:05.873205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:05.873205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:05.873205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:05.873205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:05.873205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:05.873205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:05.873623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.873623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:05.873623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.873623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:05.873623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:05.873623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:05.873623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:05.873623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:05.873623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.873623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:05.873623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:05.873623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.873623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:05.873623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:05.873623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.874097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:05.874097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:05.874097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.874097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:05.874097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:05.874097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:05.874097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:05.874097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.874097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:05.874097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:05.874097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:05.874097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.874097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:05.874097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:05.874097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:05.874097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:05.874097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:05.874097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:05.874097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:05.876129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:05.876129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:05.876129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:05.876129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:05.876129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:05.876129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:05.876129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.876129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:05.885888 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:05.889213 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:05.889213 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:05.889213 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:05.889213 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:05.889213 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:05.889213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.889655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:05.889655 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:05.889655 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:05.889655 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:05.889655 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:05.889655 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:05.889655 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.889655 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:05.889655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.889655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:05.889655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:05.889655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:05.889655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.889655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:05.889655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:05.890124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.890124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:05.890124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:05.890124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:05.890124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.890124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:05.890124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:05.890124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:05.890124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:05.890124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:05.890124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:05.890124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:05.890124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:05.890124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:05.890567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:05.890567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:05.890567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:05.890567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:05.890567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.890567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:05.890567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.890567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:05.890567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:05.890567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:05.890567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:05.890567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:05.890567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.890567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:05.890997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:05.891577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:05.891577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:05.891577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:05.891577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:05.891577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:05.891577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:05.891577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:05.891577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:05.891577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:05.891577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:08.769135 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:08.769514 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:08.769514 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:08.770063 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:08.799437 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:08.799673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:08.800524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:08.800524 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:08.800524 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:08.800524 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:08.800524 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:08.800524 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:08.800524 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:08.800524 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:08.802054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:08.802054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:08.802054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:08.802054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:08.802054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:08.802054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:08.802054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:08.802054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:08.802054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:08.802054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:08.802054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:08.802054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:08.802586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:08.802586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:08.802586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:08.802586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:08.802586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:08.802586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:08.802586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:08.802586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:08.802586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:08.802586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:08.802586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:08.802586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:08.802586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:08.802847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:08.802847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:08.803326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:08.803326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:08.803326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:08.803326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:08.803326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:08.803326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:08.803326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:08.803326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:08.803326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:08.803326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:08.803326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:08.803326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:08.803326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:08.803326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:08.803326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:08.803326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:08.803710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:08.803710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:08.803710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:08.803710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:08.803710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:08.803710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:08.803710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:08.803710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:08.803710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:08.803710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:08.803710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:08.803710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:08.803710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:08.803710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:08.803710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:08.803710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:08.803710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:08.803710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:08.803710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:08.804209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:08.804209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:08.804209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:10.849262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:10.849810 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:10.849810 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:10.850341 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:10.850505 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:10.850629 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:10.850726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:10.850823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:10.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:10.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:10.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:10.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:10.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:10.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.851212 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:10.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:10.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:10.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:10.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:10.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:10.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:10.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:10.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:10.851986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.852363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:10.852363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:10.852363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:10.852363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:10.852363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:10.852363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:10.852363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:10.852363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:10.852363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:10.852363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:10.852363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:10.852363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:10.852363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:10.852718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.852718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:10.853122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.853122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:10.853122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:10.853122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:10.853122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:10.853122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:10.853122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.853122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:10.853122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:10.853122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.853122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:10.853122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:10.853122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.853122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:10.853122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:10.853122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.853483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:10.853483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:10.853483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.853483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:10.853483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:10.853483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:10.853483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.853483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:10.853483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:10.853483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:10.853483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:10.853483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:10.853483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:10.853483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:10.853483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:10.853483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:10.853483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:10.853483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:10.853483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:10.853806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:10.853806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.853806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:10.861477 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:10.861689 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:10.861813 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:10.861945 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:39:10.862298 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:10.862490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.862946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:10.862946 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:10.862946 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:10.862946 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:10.862946 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:10.862946 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:10.862946 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.862946 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:10.863684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.863684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:10.863684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:10.863684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:10.863684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.863684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:10.863684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:10.863684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:10.863684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.863684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:10.863684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:10.863684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:10.863684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:10.863684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:10.863684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:10.864062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:10.864062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:10.864062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:10.864062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:10.864062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:10.864062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:10.864062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:10.864062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.864062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:10.864531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.864531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:10.864531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:10.864531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:10.864531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.864531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:10.864531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:10.864531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:39:10.864531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:10.864531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:10.864531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.864531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:10.864851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:10.864851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:10.864851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.864851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:10.864851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:10.864851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:10.864851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:10.864851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:10.864851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:10.864851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:10.864851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:10.864851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:10.864851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:10.864851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:10.865281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:10.865281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:10.865281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.865281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:10.865625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.865625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:10.865625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:10.865625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:10.865625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:10.865625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:10.865625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.865625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:10.865625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:10.865625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.865625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:10.865625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:10.865625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.865625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:10.865625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:10.865625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.865924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:10.865924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:10.865924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:39:10.865924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:10.865924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:10.865924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.865924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:10.865924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:10.865924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:10.865924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.865924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:10.865924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:10.865924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:10.865924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:10.865924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:10.865924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:10.865924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:10.865924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:10.866238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:10.866238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:10.866238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:10.866238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:10.866238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:10.866238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.866238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:10.869662 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:10.870041 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:10.870130 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:10.870239 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:10.870615 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:10.870873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.871420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:10.871420 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:10.871420 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:10.871420 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:10.871420 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:10.871420 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:10.871420 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.871420 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:10.871998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.871998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:10.871998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:10.871998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:10.871998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.871998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:10.871998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:10.871998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:10.871998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.871998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:10.871998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:10.871998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:10.871998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:10.871998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:10.871998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:10.872413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:10.872413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:10.872413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:10.872413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:10.872413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:10.872413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:10.872413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:10.872413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.872413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:10.872744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.872744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:10.872744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:10.872744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:10.872744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.872744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:10.872744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:10.872744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:10.872744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:10.872744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.872744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:10.873063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:10.873063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:10.873063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.873063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:10.873063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:10.873063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:10.873063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:10.873063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:10.873063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:10.873063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:10.873063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:10.873063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:10.873063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:10.873063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:10.873461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:10.873461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:10.873461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.873461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:10.873651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.873651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:10.873651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:10.873651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:10.873651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:10.873651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:10.873651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.873651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:10.873651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:10.873651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.873651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:10.873651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:10.873651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.873651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:10.873651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:10.873651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.873651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:10.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:10.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:10.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:10.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:10.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:10.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:10.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:10.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:10.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:10.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:10.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:10.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:10.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:10.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:10.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:10.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:10.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:10.874639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:10.874639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:10.874639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.874639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:10.886304 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:10.886531 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:10.886638 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:10.886791 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:10.888205 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:10.888205 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:10.888205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.888378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:10.888378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:10.888378 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:10.888378 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:10.888378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:10.888378 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:10.888378 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.888378 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:10.889240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.889240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:10.889240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:10.889240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:10.889240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.889240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:10.889240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:10.889240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.889240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:10.889240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:10.889240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:10.889240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.889643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:10.889643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:10.889643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:10.889643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:10.889643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:10.889643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:10.889643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:10.889643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:10.889643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:10.889643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:10.889643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:10.889643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:10.889643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:10.889993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.889993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:10.890107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.890107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:10.890107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:10.890107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:10.890107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:10.890107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:10.890107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.890107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:10.890107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:10.890107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.890107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:10.890107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:10.890107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.890107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:10.890107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:10.890107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:10.890574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:10.890574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:10.890574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.890574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:10.890574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:10.890574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:10.890574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:10.890574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:10.890574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:10.890574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:10.890574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:10.890574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:10.890574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:10.890574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:10.890574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:10.890574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:10.890574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:10.890574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:10.890574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:10.891101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:10.891101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:10.891101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:13.769551 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:13.770080 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:13.770080 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:13.770771 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:13.805966 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:13.806372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:13.806841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:13.806841 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:13.806841 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:13.806841 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:13.806841 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:13.806841 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:13.806841 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:13.806841 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:13.808595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:13.808595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:13.808595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:13.808595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:13.808595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:13.808595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:13.808595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:13.808595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:13.808595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:13.808595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:13.808595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:13.808595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:13.808858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:13.808858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:13.808858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:13.808858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:13.808858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:13.808858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:13.808858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:13.808858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:13.808858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:13.808858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:13.808858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:13.808858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:13.808858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:13.809078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:13.809078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:13.809078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:13.809078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:13.809078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:13.809078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:13.809078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:13.809078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:13.809078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:13.809078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:13.809078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:13.809078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:13.809078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:13.809078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:13.809078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:13.809392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:13.809392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:13.809392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:13.809392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:13.809392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:13.809392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:13.809392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:13.809392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:13.809392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:13.809392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:13.809392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:13.809392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:13.809392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:13.809392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:13.809392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:13.809392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:13.809392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:13.809392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:13.809392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:13.809724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:13.809724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:13.809724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:13.809724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:13.809724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:13.809724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:15.849678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:15.850097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:15.850097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:15.850975 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:15.851076 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:15.851217 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:15.851358 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:15.851475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:15.851582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.852165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:15.852165 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:15.852165 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:15.852165 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:15.852165 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:15.852165 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:15.852165 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.852165 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:15.852887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.852887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:15.852887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:15.852887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:15.852887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.852887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:15.852887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:15.852887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.852887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:15.852887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:15.852887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:15.852887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.853150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:15.853150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:15.853150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:15.853150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:15.853150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:15.853150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:15.853150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:15.853150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:15.853150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:15.853150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:15.853150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:15.853150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:15.853150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:15.853407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.853407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:15.853693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.853693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:15.853693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:15.853693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:15.853693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:15.853693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:15.853693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.853693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:15.853693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:15.853693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.853693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:15.853693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:15.853693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.853693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:15.853693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:15.853693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.853996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:15.853996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:15.853996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.853996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:15.853996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:15.853996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:15.853996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.853996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:15.853996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:15.853996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:15.853996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:15.853996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:15.853996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:15.853996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:15.853996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:15.853996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:15.853996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:15.853996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:15.853996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:15.854441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:15.854441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.854441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:15.861861 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:15.862078 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:15.862194 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:15.862271 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:39:15.865148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:15.865148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.865318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:15.865318 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:15.865318 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:15.865318 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:15.865318 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:15.865318 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:15.865318 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.865318 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:15.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:15.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:15.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:15.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:15.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:15.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:15.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:15.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:15.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:15.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:15.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:15.865945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:15.866582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:15.866582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:15.866582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:15.866582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:15.866582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:15.866582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:15.866582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:15.866582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.866582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:15.866582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.866582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:15.866582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:15.866582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:15.866582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.867429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:15.867429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:15.867429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:39:15.867429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:15.867429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:15.867429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.867429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:15.867429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:15.867429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:15.867429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.867429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:15.867429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:15.867429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:15.867814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:15.867814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:15.867814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:15.867814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:15.867814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:15.867814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:15.867814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:15.867814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:15.867814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:15.867814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:15.867814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.867814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:15.868012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.868012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:15.868012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:15.868012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:15.868012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:15.868012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:15.868012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.868012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:15.868012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:15.868012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.868012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:15.868012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:15.868012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.868012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:15.868012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:15.868012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.868273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:15.868273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:15.868273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:39:15.868273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:15.868273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:15.868273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.868273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:15.868273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:15.868273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:15.868273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.868273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:15.868273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:15.868273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:15.868273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:15.868273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:15.868273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:15.868273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:15.868273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:15.868547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:15.868547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:15.868547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:15.868547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:15.868547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:15.868547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.868547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:15.869974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:15.870142 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:15.870225 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:15.870346 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:15.871291 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:15.871818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.871934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:15.871934 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:15.871934 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:15.871934 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:15.871934 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:15.871934 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:15.871934 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.871934 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:15.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:15.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:15.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:15.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:15.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:15.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:15.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:15.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:15.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:15.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:15.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:15.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:15.872958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:15.872958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:15.872958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:15.872958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:15.872958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:15.872958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:15.872958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:15.872958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.872958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:15.873118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.873118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:15.873118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:15.873118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:15.873118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.873118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:15.873118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:15.873118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:15.873118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:15.873118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.873118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:15.873357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:15.873357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:15.873357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.873357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:15.873357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:15.873357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:15.873357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:15.873357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:15.873357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:15.873357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:15.873357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:15.873357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:15.873357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:15.873357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:15.873631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:15.873631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:15.873631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.873631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:15.873746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.873746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:15.873746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:15.873746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:15.873746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:15.873746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:15.873746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.873746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:15.873746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:15.873746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.873746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:15.873746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:15.873746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.873746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:15.873746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:15.873746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.873746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:15.874117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:15.874117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:15.874117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:15.874117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.874117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:15.874117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:15.874117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:15.874117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.874117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:15.874117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:15.874117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:15.874117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:15.874117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:15.874117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:15.874117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:15.874117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:15.874117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:15.874117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:15.874117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:15.874542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:15.874542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:15.874542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.874542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:15.886673 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:15.887471 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:15.887560 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:15.887710 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:15.888055 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:15.888322 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:15.888542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.891439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:15.891439 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:15.891439 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:15.891439 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:15.891439 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:15.891439 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:15.891439 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.891439 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:15.892096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.892096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:15.892096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:15.892096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:15.892096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.892096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:15.892096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:15.892096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.892096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:15.892096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:15.892096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:15.892096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.892353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:15.892353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:15.892353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:15.892353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:15.892353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:15.892353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:15.892353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:15.892353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:15.892353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:15.892353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:15.892353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:15.892353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:15.892353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:15.892578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.892578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:15.892777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.892777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:15.892777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:15.892777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:15.892777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:15.892777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:15.892777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.892777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:15.892777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:15.892777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.892777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:15.892777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:15.892777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.892777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:15.892777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:15.892777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:15.893047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:15.893047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:15.893047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.893047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:15.893047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:15.893047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:15.893047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:15.893047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:15.893047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:15.893047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:15.893047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:15.893047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:15.893047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:15.893047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:15.893047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:15.893047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:15.893047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:15.893047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:15.893047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:15.893388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:15.893388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:15.893388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:18.769969 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:18.770456 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:18.770456 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:18.771054 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:18.800310 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:18.800310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:18.800512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:18.800512 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:18.800512 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:18.800512 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:18.800512 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:18.800512 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:18.800512 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:18.800512 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:18.801146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:18.801146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:18.801146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:18.801146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:18.801146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:18.801146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:18.801146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:18.801146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:18.801146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:18.801146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:18.801146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:18.801146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:18.801455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:18.801455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:18.801455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:18.801455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:18.801455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:18.801455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:18.801455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:18.801455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:18.801455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:18.801455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:18.801455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:18.801455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:18.801455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:18.801688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:18.801688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:18.801864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:18.801864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:18.801864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:18.801864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:18.801864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:18.801864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:18.801864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:18.801864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:18.801864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:18.801864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:18.801864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:18.801864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:18.801864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:18.801864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:18.801864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:18.801864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:18.802171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:18.802171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:18.802171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:18.802171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:18.802171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:18.802171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:18.802171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:18.802171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:18.802171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:18.802171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:18.802171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:18.802171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:18.802171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:18.802171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:18.802171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:18.802171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:18.802171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:18.802171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:18.802171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:18.802514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:18.802514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:18.802514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:20.850177 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:20.850588 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:20.850588 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:20.851316 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:20.851316 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:20.851491 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:20.851491 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:20.851641 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:20.851721 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:20.851830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.852260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:20.852260 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:20.852260 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:20.852260 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:20.852260 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:20.852260 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:20.852260 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.852260 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:20.852998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.852998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:20.852998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:20.852998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:20.852998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.852998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:20.852998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:20.852998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.852998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:20.852998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:20.852998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:20.852998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.853225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:20.853225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:20.853225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:20.853225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:20.853225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:20.853225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:20.853225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:20.853225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:20.853225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:20.853225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:20.853225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:20.853225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:20.853225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:20.853493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.853493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:20.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:20.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:20.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:20.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:20.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:20.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:20.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:20.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:20.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:20.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:20.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:20.853890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.854206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:20.854206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:20.854206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.854206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:20.854206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:20.854206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:20.854206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.854206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:20.854206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:20.854206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:20.854206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:20.854206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:20.854206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:20.854206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:20.854206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:20.854206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:20.854206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:20.854206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:20.854206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:20.854611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:20.854611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.854611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:20.862163 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:20.862945 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:20.862945 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:20.862945 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:39:20.863941 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:20.864611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.864879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:20.864879 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:20.864879 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:20.864879 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:20.864879 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:20.864879 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:20.864879 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.864879 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:20.865713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.865713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:20.865713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:20.865713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:20.865713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.865713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:20.865713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:20.865713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:20.865713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.865713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:20.865713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:20.865713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:20.865713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:20.865713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:20.865713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:20.866001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:20.866001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:20.866001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:20.866001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:20.866001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:20.866001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:20.866001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:20.866001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.866001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:20.866523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.866523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:20.866523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:20.866523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:20.866523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.866523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:20.866523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:20.866523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:39:20.866523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:20.866523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:20.866523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.866523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:20.866771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:20.866771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:20.866771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.866771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:20.866771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:20.866771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:20.866771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:20.866771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:20.866771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:20.866771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:20.866771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:20.866771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:20.866771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:20.866771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:20.866982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:20.866982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:20.866982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.866982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:20.867520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.867520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:20.867520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:20.867520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:20.867520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:20.867520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:20.867520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.867520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:20.867520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:20.867520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.867520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:20.867520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:20.867520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.867520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:20.867520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:20.867520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.867863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:20.867863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:20.867863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:39:20.867863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:20.867863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:20.867863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.867863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:20.867863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:20.867863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:20.867863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.867863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:20.867863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:20.867863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:20.867863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:20.867863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:20.867863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:20.867863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:20.867863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:20.868159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:20.868159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:20.868159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:20.868159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:20.868159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:20.868159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.868159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:20.871458 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:20.871458 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:20.871458 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:20.871458 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:20.872138 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:20.872441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.873016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:20.873016 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:20.873016 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:20.873016 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:20.873016 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:20.873016 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:20.873016 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.873016 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:20.874269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.874269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:20.874269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:20.874269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:20.874269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.874269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:20.874269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:20.874269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:20.874269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.874269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:20.874269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:20.874269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:20.874269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:20.874269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:20.874269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:20.875127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:20.875127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:20.875127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:20.875127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:20.875127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:20.875127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:20.875127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:20.875127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.875127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:20.875127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.875127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:20.875127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:20.875127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:20.875127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.875675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:20.875675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:20.875675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:20.875675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:20.875675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.875675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:20.875675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:20.875675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:20.875675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.875675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:20.875675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:20.875675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:20.875675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:20.876056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:20.876056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:20.876056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:20.876056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:20.876056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:20.876056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:20.876056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:20.876056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:20.876056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:20.876056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.876056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:20.876410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.876410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:20.876410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:20.876410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:20.876410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:20.876410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:20.876410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.876410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:20.876410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:20.876410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.876410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:20.876410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:20.876410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.876410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:20.876410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:20.876410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.876410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:20.876916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:20.876916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:20.876916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:20.876916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.876916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:20.876916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:20.876916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:20.876916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.876916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:20.876916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:20.876916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:20.876916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:20.876916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:20.876916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:20.876916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:20.876916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:20.876916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:20.876916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:20.876916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:20.877463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:20.877463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:20.877463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.877463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:20.887589 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:20.889416 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:20.889416 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:20.889416 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:20.889950 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:20.891084 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:20.891674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.893031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:20.893031 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:20.893031 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:20.893031 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:20.893031 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:20.893031 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:20.893031 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.893031 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:20.894275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.894275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:20.894275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:20.894275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:20.894275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.894275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:20.894275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:20.894275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.894275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:20.894275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:20.894275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:20.894275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.894680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:20.894680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:20.894680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:20.894680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:20.894680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:20.894680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:20.894680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:20.894680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:20.894680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:20.894680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:20.894680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:20.894680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:20.894680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:20.895133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.895133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:20.895432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.895432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:20.895432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:20.895432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:20.895432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:20.895432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:20.895432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.895432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:20.895432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:20.895432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.895432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:20.895432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:20.895432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.895432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:20.895432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:20.895432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:20.895943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:20.895943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:20.895943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.895943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:20.895943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:20.895943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:20.895943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:20.895943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:20.895943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:20.895943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:20.895943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:20.895943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:20.895943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:20.895943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:20.895943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:20.895943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:20.895943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:20.895943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:20.895943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:20.896357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:20.896357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:20.896357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:23.770430 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:23.772515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:23.772515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:23.772515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:23.798772 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:23.798956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:23.801035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:23.801035 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:23.801035 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:23.801035 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:23.801035 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:23.801035 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:23.801035 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:23.801035 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:23.801035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:23.801035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:23.801035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:23.801035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:23.801035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:23.801035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:23.801035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:23.801481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:23.801481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:23.801481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:23.801481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:23.801481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:23.801481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:23.801481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:23.801481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:23.801481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:23.801481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:23.801481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:23.801481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:23.801481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:23.801481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:23.801781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:23.801781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:23.801781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:23.801781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:23.801781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:23.801781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:23.801781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:23.801781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:23.801781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:23.801781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:23.801781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:23.801781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:23.801781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:23.801781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:23.802107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:23.802529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:23.802529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:23.802529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:23.802529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:23.802529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:23.802529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:23.802529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:23.802529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:23.802529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:25.850589 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:25.852141 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:25.852141 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:25.852569 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:25.852781 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:25.852897 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:25.853009 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:25.853106 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:25.853207 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:25.853342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.855920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:25.855920 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:25.855920 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:25.855920 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:25.855920 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:25.855920 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:25.855920 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.855920 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:25.855920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.855920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:25.855920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:25.855920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:25.855920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.855920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:25.855920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:25.856358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.856358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:25.856358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:25.856358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:25.856358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.856358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:25.856358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:25.856358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:25.856358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:25.856358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:25.856358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:25.856358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:25.856358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:25.856358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:25.856717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:25.856717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:25.856717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:25.856717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:25.856717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.856717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:25.856717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.856717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:25.856717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:25.856717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:25.856717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:25.856717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:25.856717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.856717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:25.857075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:25.857592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:25.857592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:25.857592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:25.857592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:25.857592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:25.857592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:25.857592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:25.857592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:25.857592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.857592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:25.862552 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:25.863130 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:25.863130 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:25.863130 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:39:25.863406 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:25.863581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.866612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:25.866612 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:25.866612 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:25.866612 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:25.866612 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:25.866612 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:25.866612 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.866612 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:25.866612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.866612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:25.866612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:25.866612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:25.866612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.866612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:25.866612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:25.866612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:25.866612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.866612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:25.867126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:25.867126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:25.867126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:25.867126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:25.867126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:25.867126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:25.867126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:25.867126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:25.867126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:25.867126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:25.867126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:25.867126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:25.867126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.867126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:25.867126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.867558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:25.867558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:25.867558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:25.867558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.867558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:25.867558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:25.867558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:39:25.867558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:25.867558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:25.867558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.867558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:25.867558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:25.867558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:25.867901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.867901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:25.867901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:25.867901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:25.867901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:25.867901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:25.867901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:25.867901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:25.867901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:25.867901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:25.867901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:25.867901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:25.867901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:25.867901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:25.868262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.868262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:25.868262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.868262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:25.868262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:25.868262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:25.868262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:25.868262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:25.868262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.868262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:25.868262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:25.868262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.868262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:25.868262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:25.868262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.868666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:25.868666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:25.868666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.868666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:25.868666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:25.868666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:39:25.868666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:25.868666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:25.868666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.868666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:25.868666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:25.868666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:25.868666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.868666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:25.868666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:25.868666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:25.868666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:25.868666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:25.868666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:25.869136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:25.869136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:25.869136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:25.869136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:25.869136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:25.869136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:25.869136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:25.869136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.869136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:25.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:25.871332 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:25.871525 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:25.871721 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:25.872099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:25.872346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.875614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:25.875614 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:25.875614 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:25.875614 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:25.875614 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:25.875614 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:25.875614 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.875614 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:25.875614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.875614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:25.875614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:25.875614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:25.875614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.875614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:25.875614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:25.875614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:25.875614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.875614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:25.876130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:25.876130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:25.876130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:25.876130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:25.876130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:25.876130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:25.876130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:25.876130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:25.876130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:25.876130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:25.876130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:25.876130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:25.876130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.876130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:25.876130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.876550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:25.876550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:25.876550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:25.876550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.876550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:25.876550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:25.876550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:25.876550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:25.876550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.876550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:25.876550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:25.876550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:25.876550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.876896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:25.876896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:25.876896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:25.876896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:25.876896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:25.876896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:25.876896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:25.876896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:25.876896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:25.876896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:25.876896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:25.876896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:25.876896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:25.877217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.877217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:25.877217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.877217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:25.877217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:25.877217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:25.877217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:25.877217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:25.877217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.877217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:25.877217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:25.877217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.877217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:25.877217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:25.877217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.877646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:25.877646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:25.877646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.877646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:25.877646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:25.877646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:25.877646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:25.877646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.877646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:25.877646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:25.877646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:25.877646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.877646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:25.877646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:25.877646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:25.877646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:25.877646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:25.877646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:25.877646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:25.878122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:25.878122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:25.878122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:25.878122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:25.878122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:25.878122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:25.878122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.878122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:25.889439 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:25.890191 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:25.890191 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:25.890191 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:25.890598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:25.891218 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:25.891405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.894124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:25.894124 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:25.894124 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:25.894124 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:25.894124 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:25.894124 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:25.894124 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.894124 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:25.894124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.894124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:25.894124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:25.894124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:25.894124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.894124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:25.894124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:25.894585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.894585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:25.894585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:25.894585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:25.894585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.894585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:25.894585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:25.894585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:25.894585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:25.894585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:25.894585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:25.894585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:25.894585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:25.894585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:25.894943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:25.894943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:25.894943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:25.894943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:25.894943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.894943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:25.894943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.894943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:25.894943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:25.894943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:25.894943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:25.894943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:25.894943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.894943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:25.895339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:25.895850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:25.895850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:25.895850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:25.895850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:25.895850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:25.895850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:25.895850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:25.895850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:25.895850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:25.895850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:28.771367 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:28.771730 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:28.771730 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:28.772295 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:28.798993 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:28.799593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:28.799752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:28.799752 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:28.799752 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:28.799752 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:28.799752 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:28.799752 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:28.799752 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:28.799752 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:28.800371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:28.800371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:28.800371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:28.800371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:28.800371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:28.800371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:28.800371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:28.800371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:28.800371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:28.800371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:28.800371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:28.800371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:28.800587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:28.800587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:28.800587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:28.800587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:28.800587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:28.800587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:28.800587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:28.800587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:28.800587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:28.800587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:28.800587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:28.800587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:28.800587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:28.800811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:28.800811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:28.801154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:28.801154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:28.801154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:28.801154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:28.801154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:28.801154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:28.801154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:28.801154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:28.801154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:28.801154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:28.801154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:28.801154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:28.801154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:28.801154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:28.801154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:28.801154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:28.801465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:28.801465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:28.801465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:28.801465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:28.801465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:28.801465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:28.801465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:28.801465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:28.801465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:28.801465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:28.801465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:28.801465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:28.801465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:28.801465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:28.801465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:28.801465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:28.801465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:28.801465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:28.801465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:28.801773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:28.801773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:28.801773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:30.851472 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:30.852197 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:30.852336 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:30.854151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:30.854151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:30.854151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:30.854151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:30.854151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:30.854409 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:30.854409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.854409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:30.854409 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:30.854409 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:30.854409 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:30.854409 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:30.854409 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:30.854409 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.854409 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:30.854952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.854952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:30.854952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:30.854952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:30.854952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.854952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:30.854952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:30.854952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.854952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:30.854952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:30.854952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:30.854952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.855146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:30.855146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:30.855146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:30.855146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:30.855146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:30.855146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:30.855146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:30.855146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:30.855146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:30.855146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:30.855146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:30.855146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:30.855146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:30.855391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.855391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:30.855707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.855707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:30.855707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:30.855707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:30.855707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:30.855707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:30.855707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.855707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:30.855707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:30.855707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.855707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:30.855707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:30.855707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.855707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:30.855707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:30.855707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.855958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:30.855958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:30.855958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.855958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:30.855958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:30.855958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:30.855958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.855958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:30.855958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:30.855958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:30.855958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:30.855958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:30.855958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:30.855958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:30.855958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:30.855958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:30.855958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:30.855958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:30.855958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:30.856226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:30.856226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.856226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:30.864339 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:30.864790 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:30.864893 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:30.864982 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:39:30.865673 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:30.865673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.865782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:30.865782 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:30.865782 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:30.865782 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:30.865782 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:30.865782 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:30.865782 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.865782 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:30.866304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.866304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:30.866304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:30.866304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:30.866304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.866304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:30.866304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:30.866304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:30.866304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.866304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:30.866304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:30.866304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:30.866304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:30.866304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:30.866304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:30.866570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:30.866570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:30.866570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:30.866570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:30.866570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:30.866570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:30.866570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:30.866570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.866570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:30.866878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.866878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:30.866878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:30.866878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:30.866878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.866878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:30.866878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:30.866878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:39:30.866878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:30.866878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:30.866878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.866878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:30.867075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:30.867075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:30.867075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.867075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:30.867075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:30.867075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:30.867075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:30.867075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:30.867075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:30.867075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:30.867075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:30.867075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:30.867075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:30.867075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:30.867374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:30.867374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:30.867374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.867374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:30.867556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.867556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:30.867556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:30.867556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:30.867556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:30.867556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:30.867556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.867556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:30.867556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:30.867556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.867556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:30.867556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:30.867556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.867556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:30.867556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:30.867556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.867819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:30.867819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:30.867819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:39:30.867819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:30.867819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:30.867819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.867819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:30.867819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:30.867819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:30.867819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.867819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:30.867819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:30.867819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:30.867819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:30.867819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:30.867819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:30.867819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:30.867819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:30.868071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:30.868071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:30.868071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:30.868071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:30.868071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:30.868071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.868071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:30.871425 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:30.871706 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:30.871800 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:30.871895 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:30.872578 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:30.872578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.872697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:30.872697 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:30.872697 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:30.872697 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:30.872697 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:30.872697 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:30.872697 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.872697 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:30.873170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.873170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:30.873170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:30.873170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:30.873170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.873170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:30.873170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:30.873170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:30.873170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.873170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:30.873170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:30.873170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:30.873170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:30.873170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:30.873170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:30.873458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:30.873458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:30.873458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:30.873458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:30.873458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:30.873458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:30.873458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:30.873458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.873458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:30.873778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.873778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:30.873778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:30.873778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:30.873778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.873778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:30.873778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:30.873778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:30.873778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:30.873778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.873778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:30.873975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:30.873975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:30.873975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.873975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:30.873975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:30.873975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:30.873975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:30.873975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:30.873975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:30.873975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:30.873975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:30.873975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:30.873975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:30.873975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:30.874188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:30.874188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:30.874188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.874188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:30.874491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.874491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:30.874491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:30.874491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:30.874491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:30.874491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:30.874491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.874491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:30.874491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:30.874491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.874491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:30.874491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:30.874491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.874491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:30.874491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:30.874491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.874491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:30.874796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:30.874796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:30.874796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:30.874796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.874796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:30.874796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:30.874796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:30.874796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.874796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:30.874796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:30.874796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:30.874796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:30.874796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:30.874796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:30.874796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:30.874796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:30.874796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:30.874796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:30.874796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:30.875082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:30.875082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:30.875082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.875082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:30.889881 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:30.890168 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:30.890279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:30.890415 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:30.891299 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:30.891299 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:30.891299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.891413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:30.891413 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:30.891413 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:30.891413 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:30.891413 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:30.891413 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:30.891413 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.891413 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:30.892000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.892000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:30.892000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:30.892000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:30.892000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.892000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:30.892000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:30.892000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.892000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:30.892000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:30.892000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:30.892000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.892211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:30.892211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:30.892211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:30.892211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:30.892211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:30.892211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:30.892211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:30.892211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:30.892211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:30.892211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:30.892211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:30.892211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:30.892211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:30.892441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.892441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:30.892694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.892694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:30.892694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:30.892694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:30.892694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:30.892694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:30.892694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.892694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:30.892694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:30.892694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.892694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:30.892694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:30.892694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.892694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:30.892694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:30.892694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:30.892936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:30.892936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:30.892936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.892936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:30.892936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:30.892936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:30.892936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:30.892936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:30.892936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:30.892936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:30.892936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:30.892936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:30.892936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:30.892936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:30.892936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:30.892936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:30.892936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:30.892936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:30.892936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:30.893219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:30.893219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:30.893219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:33.771768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:33.772389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:33.772389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:33.773075 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:33.805742 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:33.805742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:33.806093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:33.806093 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:33.806093 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:33.806093 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:33.806093 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:33.806093 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:33.806093 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:33.806093 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:33.807185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:33.807185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:33.807185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:33.807185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:33.807185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:33.807185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:33.807185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:33.807185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:33.807185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:33.807185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:33.807185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:33.807185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:33.807711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:33.807711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:33.807711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:33.807711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:33.807711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:33.807711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:33.807711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:33.807711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:33.807711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:33.807711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:33.807711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:33.807711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:33.807711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:33.808128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:33.808128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:33.808910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:33.808910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:33.808910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:33.808910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:33.808910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:33.808910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:33.808910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:33.808910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:33.808910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:33.808910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:33.808910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:33.808910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:33.808910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:33.808910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:33.808910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:33.808910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:33.809588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:33.809588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:33.809588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:33.809588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:33.809588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:33.809588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:33.809588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:33.809588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:33.809588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:33.809588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:33.809588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:33.809588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:33.809588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:33.809588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:33.809588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:33.809588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:33.809588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:33.809588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:33.809588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:33.810269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:33.810269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:33.810269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:35.852273 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:35.852811 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:35.852811 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:35.853494 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:35.853700 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:35.853820 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:35.853924 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:35.854039 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:35.854145 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:35.854281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.854767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:35.854767 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:35.854767 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:35.854767 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:35.854767 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:35.854767 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:35.854767 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.854767 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:35.855562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.855562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:35.855562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:35.855562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:35.855562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.855562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:35.855562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:35.855562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.855562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:35.855562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:35.855562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:35.855562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.855855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:35.855855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:35.855855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:35.855855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:35.855855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:35.855855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:35.855855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:35.855855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:35.855855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:35.855855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:35.855855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:35.855855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:35.855855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:35.856083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.856083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:35.856353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.856353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:35.856353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:35.856353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:35.856353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:35.856353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:35.856353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.856353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:35.856353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:35.856353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.856353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:35.856353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:35.856353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.856353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:35.856353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:35.856353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.856769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:35.856769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:35.856769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.856769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:35.856769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:35.856769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:35.856769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.856769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:35.856769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:35.856769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:35.856769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:35.856769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:35.856769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:35.856769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:35.856769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:35.856769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:35.856769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:35.856769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:35.856769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:35.864791 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:35.865010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:35.865114 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:35.865234 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:39:35.865747 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:35.865885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.866380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:35.866380 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:35.866380 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:35.866380 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:35.866380 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:35.866380 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:35.866380 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.866380 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:35.867287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.867287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:35.867287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:35.867287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:35.867287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.867287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:35.867287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:35.867287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:35.867287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.867287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:35.867287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:35.867287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:35.867287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:35.867287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:35.867287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:35.867747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:35.867747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:35.867747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:35.867747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:35.867747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:35.867747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:35.867747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:35.867747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.867747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:35.868235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.868235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:35.868235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:35.868235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:35.868235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.868235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:35.868235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:35.868235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:39:35.868235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:35.868235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:35.868235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.868235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:35.868822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:35.868822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:35.868822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.868822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:35.868822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:35.868822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:35.868822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:35.868822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:35.868822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:35.868822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:35.868822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:35.868822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:35.868822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:35.868822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:35.869131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:35.869131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:35.869131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.869131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:35.869131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.869131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:35.869131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:35.869131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:35.869131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:35.869131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:35.869131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.869131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:35.869131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:35.869131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.869454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:35.869454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:35.869454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.869454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:35.869454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:35.869454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.869454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:35.869454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:35.869454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:39:35.869454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:35.869454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:35.869454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.869454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:35.869454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:35.869454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:35.869454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.869454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:35.869454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:35.869454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:35.869836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:35.869836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:35.869836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:35.869836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:35.869836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:35.869836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:35.869836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:35.869836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:35.869836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:35.869836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:35.869836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.869836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:35.871695 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:35.871882 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:35.871992 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:35.872112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:35.872517 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:35.872756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.873176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:35.873176 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:35.873176 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:35.873176 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:35.873176 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:35.873176 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:35.873176 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.873176 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:35.873959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.873959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:35.873959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:35.873959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:35.873959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.873959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:35.873959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:35.873959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:35.873959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.873959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:35.873959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:35.873959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:35.873959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:35.873959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:35.873959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:35.874448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:35.874448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:35.874448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:35.874448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:35.874448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:35.874448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:35.874448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:35.874448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.874448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:35.874853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.874853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:35.874853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:35.874853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:35.874853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.874853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:35.874853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:35.874853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:35.874853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:35.874853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.874853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:35.875402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:35.875402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:35.875402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.875402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:35.875402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:35.875402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:35.875402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:35.875402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:35.875402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:35.875402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:35.875402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:35.875402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:35.875402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:35.875402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:35.876044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:35.876044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:35.876044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.876044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:35.876044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.876044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:35.876044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:35.876044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:35.876044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:35.876044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:35.876044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.876044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:35.876044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:35.876044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.876776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:35.876776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:35.876776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.876776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:35.876776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:35.876776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.876776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:35.876776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:35.876776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:35.876776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:35.876776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.876776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:35.876776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:35.876776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:35.876776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.876776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:35.876776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:35.876776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:35.876776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:35.877191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:35.877191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:35.877191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:35.877191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:35.877191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:35.877191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:35.877191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:35.877191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:35.877191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:35.877191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.877191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:35.890192 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:35.890410 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:35.890519 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:35.890677 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:35.891160 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:35.891419 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:35.891753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.892242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:35.892242 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:35.892242 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:35.892242 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:35.892242 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:35.892242 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:35.892242 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.892242 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:35.893125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.893125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:35.893125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:35.893125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:35.893125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.893125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:35.893125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:35.893125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.893125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:35.893125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:35.893125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:35.893125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.893667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:35.893667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:35.893667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:35.893667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:35.893667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:35.893667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:35.893667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:35.893667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:35.893667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:35.893667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:35.893667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:35.893667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:35.893667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:35.893994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.893994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:35.894130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.894130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:35.894130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:35.894130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:35.894130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:35.894130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:35.894130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.894130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:35.894130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:35.894130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.894130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:35.894130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:35.894130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.894130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:35.894130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:35.894130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:35.894453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:35.894453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:35.894453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.894453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:35.894453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:35.894453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:35.894453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:35.894453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:35.894453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:35.894453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:35.894453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:35.894453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:35.894453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:35.894453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:35.894453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:35.894453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:35.894453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:35.894453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:35.894453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:35.894798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:35.894798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:35.894798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:38.772191 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:38.772699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:38.772699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:38.773505 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:38.802354 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:38.802543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:38.803001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:38.803001 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:38.803001 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:38.803001 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:38.803001 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:38.803001 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:38.803001 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:38.803001 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:38.803681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:38.803681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:38.803681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:38.803681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:38.803681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:38.803681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:38.803681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:38.803681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:38.803681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:38.803681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:38.803681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:38.803681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:38.803942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:38.803942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:38.803942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:38.803942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:38.803942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:38.803942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:38.803942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:38.803942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:38.803942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:38.803942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:38.803942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:38.803942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:38.803942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:38.804198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:38.804198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:38.804376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:38.804376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:38.804376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:38.804376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:38.804376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:38.804376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:38.804376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:38.804376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:38.804376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:38.804376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:38.804376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:38.804376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:38.804376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:38.804376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:38.804376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:38.804376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:38.804674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:38.804674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:38.804674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:38.804674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:38.804674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:38.804674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:38.804674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:38.804674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:38.804674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:38.804674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:38.804674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:38.804674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:38.804674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:38.804674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:38.804674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:38.804674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:38.804674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:38.804674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:38.804674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:38.805026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:38.805026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:38.805026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:40.852685 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:40.854233 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:40.854233 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:40.854705 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:40.854884 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:40.854995 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:40.855103 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:40.855197 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:40.855333 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:40.855433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.858194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:40.858194 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:40.858194 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:40.858194 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:40.858194 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:40.858194 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:40.858194 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.858194 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:40.858194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.858194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:40.858194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:40.858194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:40.858194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.858194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:40.858194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:40.858717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.858717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:40.858717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:40.858717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:40.858717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.858717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:40.858717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:40.858717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:40.858717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:40.858717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:40.858717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:40.858717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:40.858717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:40.858717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:40.859110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:40.859110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:40.859110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:40.859110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:40.859110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.859110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:40.859110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.859110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:40.859110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:40.859110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:40.859110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:40.859110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:40.859110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.859110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:40.859492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:40.860085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:40.860085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:40.860085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:40.860085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:40.860085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:40.860085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:40.860085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:40.860085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:40.860085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.860085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:40.865075 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:40.865859 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:40.865859 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:40.865859 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:39:40.866166 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:40.866355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.868779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:40.868779 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:40.868779 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:40.868779 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:40.868779 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:40.868779 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:40.868779 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.868779 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:40.868779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.868779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:40.868779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:40.868779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:40.868779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.868779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:40.868779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:40.868779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:40.868779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.868779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:40.869117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:40.869117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:40.869117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:40.869117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:40.869117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:40.869117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:40.869117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:40.869117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:40.869117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:40.869117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:40.869117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:40.869117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:40.869117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.869117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:40.869117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.869434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:40.869434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:40.869434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:40.869434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.869434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:40.869434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:40.869434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:39:40.869434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:40.869434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:40.869434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.869434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:40.869434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:40.869434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:40.869703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.869703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:40.869703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:40.869703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:40.869703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:40.869703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:40.869703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:40.869703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:40.869703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:40.869703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:40.869703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:40.869703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:40.869703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:40.869703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:40.869933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.869933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:40.869933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.869933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:40.869933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:40.869933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:40.869933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:40.869933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:40.869933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.869933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:40.869933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:40.869933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.869933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:40.869933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:40.869933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.870177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:40.870177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:40.870177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.870177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:40.870177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:40.870177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:39:40.870177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:40.870177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:40.870177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.870177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:40.870177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:40.870177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:40.870177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.870177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:40.870177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:40.870177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:40.870177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:40.870177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:40.870177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:40.870534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:40.870534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:40.870534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:40.870534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:40.870534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:40.870534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:40.870534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:40.870534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.870534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:40.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:40.872516 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:40.872516 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:40.872516 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:40.872857 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:40.873056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.875305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:40.875305 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:40.875305 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:40.875305 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:40.875305 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:40.875305 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:40.875305 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.875305 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:40.875305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.875305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:40.875305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:40.875305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:40.875305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.875305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:40.875305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:40.875305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:40.875305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.875305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:40.875667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:40.875667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:40.875667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:40.875667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:40.875667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:40.875667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:40.875667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:40.875667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:40.875667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:40.875667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:40.875667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:40.875667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:40.875667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.875667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:40.875667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.875985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:40.875985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:40.875985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:40.875985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.875985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:40.875985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:40.875985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:40.875985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:40.875985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.875985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:40.875985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:40.875985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:40.875985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.876288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:40.876288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:40.876288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:40.876288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:40.876288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:40.876288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:40.876288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:40.876288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:40.876288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:40.876288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:40.876288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:40.876288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:40.876288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:40.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:40.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:40.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:40.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:40.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:40.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:40.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:40.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:40.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:40.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:40.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.876772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:40.876772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:40.876772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.876772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:40.876772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:40.876772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:40.876772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:40.876772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.876772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:40.876772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:40.876772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:40.876772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.876772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:40.876772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:40.876772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:40.876772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:40.876772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:40.876772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:40.876772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:40.877060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:40.877060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:40.877060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:40.877060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:40.877060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:40.877060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:40.877060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.877060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:40.890498 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:40.891109 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:40.891109 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:40.891109 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:40.891494 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:40.891780 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:40.891932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.893807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:40.893807 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:40.893807 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:40.893807 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:40.893807 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:40.893807 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:40.893807 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.893807 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:40.893807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.893807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:40.893807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:40.893807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:40.893807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.893807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:40.893807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:40.894123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.894123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:40.894123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:40.894123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:40.894123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.894123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:40.894123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:40.894123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:40.894123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:40.894123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:40.894123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:40.894123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:40.894123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:40.894123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:40.894358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:40.894358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:40.894358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:40.894358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:40.894358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.894358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:40.894358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.894358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:40.894358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:40.894358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:40.894358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:40.894358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:40.894358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.894358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:40.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:40.895384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:40.895384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:40.895384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:40.895384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:40.895384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:40.895384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:40.895384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:40.895384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:40.895384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:40.895384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:43.772656 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:43.774866 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:43.774866 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:43.774866 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:43.804029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:43.804247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:43.806652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:43.806652 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:43.806652 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:43.806652 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:43.806652 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:43.806652 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:43.806652 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:43.806652 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:43.806652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:43.806652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:43.806652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:43.806652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:43.806652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:43.806652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:43.806652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:43.807294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:43.807294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:43.807294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:43.807294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:43.807294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:43.807294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:43.807294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:43.807294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:43.807294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:43.807294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:43.807294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:43.807294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:43.807294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:43.807294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:43.807791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:43.807791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:43.807791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:43.807791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:43.807791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:43.807791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:43.808193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:43.808193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:43.808193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:43.808193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:43.808193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:43.808193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:43.808193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:43.808193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:43.808193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:43.808193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:43.808193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:43.808193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:43.808193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:43.808193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:43.808193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:43.808193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:43.808781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:43.808781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:43.808781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:43.808781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:43.808781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:43.808781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:43.808781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:43.808781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:43.808781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:43.808781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:43.808781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:43.808781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:43.808781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:43.808781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:43.808781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:43.808781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:43.808781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:43.808781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:43.808781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:43.809363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:43.809363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:43.809363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:45.853576 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:45.854993 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:45.854993 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:45.855264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:45.855491 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:45.855639 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:45.855756 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:45.855911 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:45.856018 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:45.856132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.858807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:45.858807 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:45.858807 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:45.858807 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:45.858807 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:45.858807 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:45.858807 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.858807 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:45.858807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.858807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:45.858807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:45.858807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:45.858807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.858807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:45.858807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:45.859296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.859296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:45.859296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:45.859296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:45.859296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.859296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:45.859296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:45.859296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:45.859296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:45.859296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:45.859296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:45.859296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:45.859296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:45.859296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:45.859619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:45.859619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:45.859619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:45.859619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:45.859619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.859619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:45.859619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.859619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:45.859619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:45.859619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:45.859619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:45.859619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:45.859619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.859619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:45.859933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:45.861142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:45.861142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:45.861142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:45.861142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:45.861142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:45.861142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:45.861142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:45.861142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:45.861142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.861142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:45.865638 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:45.866450 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:45.866450 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:45.866450 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:39:45.866770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:45.866890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.867383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:45.867383 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:45.867383 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:45.867383 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:45.867383 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:45.867383 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:45.867383 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.867383 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:45.870583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.870583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:45.870583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:45.870583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:45.870583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.870583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:45.870583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:45.870583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:45.870583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.870583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:45.870583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:45.870583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:45.870583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:45.870583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:45.870583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:45.871532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:45.871532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:45.871532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:45.871532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:45.871532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:45.871532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:45.871532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:45.871532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.871532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:45.871532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.871532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:45.871532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:45.871532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:45.871532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.871990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:45.871990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:45.871990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:39:45.871990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:45.871990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:45.871990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.871990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:45.871990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:45.871990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:45.871990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.871990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:45.871990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:45.871990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:45.872211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:45.872211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:45.872211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:45.872211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:45.872211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:45.872211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:45.872211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:45.872211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:45.872211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:45.872211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:45.872211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.872211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:45.872429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.872429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:45.872429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:45.872429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:45.872429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:45.872429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:45.872429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.872429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:45.872429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:45.872429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.872429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:45.872429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:45.872429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.872429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:45.872429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:45.872429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.872686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:45.872686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:45.872686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:39:45.872686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:45.872686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:45.872686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.872686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:45.872686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:45.872686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:45.872686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.872686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:45.872686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:45.872686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:45.872686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:45.872686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:45.872686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:45.872686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:45.872686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:45.872971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:45.872971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:45.872971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:45.872971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:45.872971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:45.872971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.872971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:45.872971 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:45.874135 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:45.874458 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:45.874758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:45.875365 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:45.875634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.878089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:45.878089 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:45.878089 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:45.878089 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:45.878089 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:45.878089 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:45.878089 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.878089 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:45.878089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.878089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:45.878089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:45.878089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:45.878089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.878089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:45.878089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:45.878089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:45.878089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.878089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:45.879226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:45.879226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:45.879226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:45.879226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:45.879226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:45.879226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:45.879226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:45.879226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:45.879226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:45.879226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:45.879226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:45.879226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:45.879226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.879226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:45.879226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.880060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:45.880060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:45.880060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:45.880060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.880060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:45.880060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:45.880060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:45.880060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:45.880060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.880060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:45.880060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:45.880060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:45.880060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.880488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:45.880488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:45.880488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:45.880488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:45.880488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:45.880488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:45.880488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:45.880488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:45.880488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:45.880488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:45.880488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:45.880488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:45.880488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:45.880700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.880700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:45.880700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.880700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:45.880700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:45.880700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:45.880700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:45.880700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:45.880700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.880700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:45.880700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:45.880700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.880700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:45.880700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:45.880700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:45.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:45.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:45.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:45.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:45.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:45.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:45.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:45.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:45.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:45.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:45.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:45.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:45.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:45.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:45.880934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:45.881233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:45.881233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:45.881233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:45.881233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:45.881233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:45.881233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:45.881233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.881233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:45.891035 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:45.891909 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:45.891909 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:45.891909 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:45.892127 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:45.892261 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:45.892466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.894261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:45.894261 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:45.894261 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:45.894261 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:45.894261 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:45.894261 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:45.894261 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.894261 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:45.894261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.894261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:45.894261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:45.894261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:45.894261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.894261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:45.894261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:45.894630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.894630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:45.894630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:45.894630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:45.894630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.894630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:45.894630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:45.894630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:45.894630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:45.894630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:45.894630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:45.894630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:45.894630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:45.894630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:45.894915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:45.894915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:45.894915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:45.894915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:45.894915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.894915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:45.894915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.894915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:45.894915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:45.894915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:45.894915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:45.894915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:45.894915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.894915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:45.895145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:45.895525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:45.895525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:45.895525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:45.895525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:45.895525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:45.895525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:45.895525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:45.895525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:45.895525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:45.895525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:48.773740 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:48.776478 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:48.776478 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:48.776478 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:48.809874 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:48.810122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:48.811588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:48.811588 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:48.811588 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:48.811588 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:48.811588 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:48.811588 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:48.811588 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:48.811588 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:48.812059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:48.812059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:48.812059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:48.812059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:48.812059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:48.812059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:48.812059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:48.812059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:48.812059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:48.812059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:48.812059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:48.812059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:48.812689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:48.812689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:48.812689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:48.812689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:48.812689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:48.812689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:48.812689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:48.812689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:48.812689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:48.812689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:48.812689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:48.812689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:48.812689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:48.814451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:48.814451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:48.814451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:48.814451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:48.814451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:48.814451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:48.814451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:48.814451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:48.814451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:48.814451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:48.814451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:48.814451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:48.814451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:48.814451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:48.814451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:48.815177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:48.815177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:48.815177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:48.815177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:48.815177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:48.815177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:48.815177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:48.815177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:48.815177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:48.815177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:48.815177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:48.815177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:48.815177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:48.815177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:48.815177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:48.815177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:48.815177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:48.815177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:48.815177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:48.815478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:48.815478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:48.815478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:48.815478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:48.815478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:48.815478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:50.854437 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:50.856167 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:50.856167 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:50.856436 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:50.856672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:50.856775 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:50.856895 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:50.857012 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:50.857147 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:50.857306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.859429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:50.859429 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:50.859429 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:50.859429 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:50.859429 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:50.859429 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:50.859429 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.859429 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:50.859429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.859429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:50.859429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:50.859429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:50.859429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.859429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:50.859429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:50.859740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.859740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:50.859740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:50.859740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:50.859740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.859740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:50.859740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:50.859740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:50.859740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:50.859740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:50.859740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:50.859740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:50.859740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:50.859740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:50.860007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:50.860007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:50.860007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:50.860007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:50.860007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.860007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:50.860007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.860007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:50.860007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:50.860007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:50.860007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:50.860007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:50.860007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.860007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:50.860306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:50.860598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:50.860598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:50.860598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:50.860598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:50.860598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:50.860598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:50.860598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:50.860598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:50.860598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.860598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:50.866116 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:50.866880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:50.866880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:50.866880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:39:50.867173 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:50.867357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.869897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:50.869897 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:50.869897 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:50.869897 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:50.869897 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:50.869897 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:50.869897 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.869897 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:50.869897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.869897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:50.869897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:50.869897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:50.869897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.869897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:50.869897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:50.869897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:50.869897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.869897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:50.870421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:50.870421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:50.870421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:50.870421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:50.870421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:50.870421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:50.870421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:50.870421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:50.870421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:50.870421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:50.870421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:50.870421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:50.870421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.870421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:50.870421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.870832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:50.870832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:50.870832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:50.870832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.870832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:50.870832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:50.870832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:39:50.870832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:50.870832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:50.870832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.870832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:50.870832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:50.870832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:50.871037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.871037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:50.871037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:50.871037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:50.871037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:50.871037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:50.871037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:50.871037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:50.871037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:50.871037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:50.871037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:50.871037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:50.871037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:50.871037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:50.871295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.871295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:50.871295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.871295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:50.871295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:50.871295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:50.871295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:50.871295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:50.871295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.871295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:50.871295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:50.871295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.871295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:50.871295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:50.871295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.871528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:50.871528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:50.871528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.871528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:50.871528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:50.871528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:39:50.871528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:50.871528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:50.871528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.871528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:50.871528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:50.871528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:50.871528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.871528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:50.871528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:50.871528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:50.871528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:50.871528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:50.871528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:50.871868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:50.871868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:50.871868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:50.871868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:50.871868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:50.871868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:50.871868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:50.871868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.871868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:50.874204 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:50.874703 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:50.875146 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:50.875146 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:50.875578 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:50.875797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.878654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:50.878654 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:50.878654 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:50.878654 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:50.878654 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:50.878654 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:50.878654 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.878654 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:50.878654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.878654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:50.878654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:50.878654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:50.878654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.878654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:50.878654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:50.878654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:50.878654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.878654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:50.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:50.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:50.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:50.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:50.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:50.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:50.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:50.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:50.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:50.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:50.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:50.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:50.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:50.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.879520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:50.879520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:50.879520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:50.879520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.879520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:50.879520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:50.879520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:50.879520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:50.879520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.879520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:50.879520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:50.879520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:50.879520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.879831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:50.879831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:50.879831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:50.879831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:50.879831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:50.879831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:50.879831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:50.879831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:50.879831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:50.879831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:50.879831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:50.879831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:50.879831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:50.880128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.880128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:50.881262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.881262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:50.881262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:50.881262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:50.881262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:50.881262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:50.881262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.881262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:50.881262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:50.881262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.881262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:50.881262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:50.881262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.881262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:50.881262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:50.881262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.881262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:50.882113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:50.882113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:50.882113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.882113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:50.891558 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:50.892286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:50.892286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:50.892286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:50.892755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:50.893027 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:50.893298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.893913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:50.893913 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:50.893913 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:50.893913 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:50.893913 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:50.893913 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:50.893913 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.893913 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:50.895766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.895766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:50.895766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:50.895766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:50.895766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.895766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:50.895766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:50.895766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.895766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:50.895766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:50.895766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:50.895766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.896482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:50.896482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:50.896482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:50.896482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:50.896482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:50.896482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:50.896482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:50.896482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:50.896482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:50.896482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:50.896482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:50.896482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:50.896482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:50.896739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.896739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:50.896739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.896739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:50.896739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:50.896739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:50.896739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:50.896739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:50.896739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.896739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:50.896739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:50.896739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.896739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:50.896739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:50.896739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.896967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:50.896967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:50.896967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:50.896967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:50.896967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:50.896967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.896967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:50.896967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:50.896967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:50.896967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:50.896967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:50.896967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:50.896967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:50.896967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:50.896967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:50.896967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:50.896967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:50.896967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:50.896967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:50.897237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:50.897237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:50.897237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:50.897237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:50.897237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:50.897237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:53.774713 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:53.776776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:53.776776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:53.776776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:53.780752 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=239 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:39:53.808226 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:53.808518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:53.809033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:53.809033 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:53.809033 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:53.809033 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:53.809033 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:53.809033 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:53.809033 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:53.809033 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:53.809952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:53.809952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:53.809952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:53.809952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:53.809952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:53.809952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:53.809952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:53.809952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:53.809952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:53.809952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:53.809952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:53.809952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:53.810511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:53.810511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:53.810511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:53.810511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:53.810511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:53.810511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:53.810511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:53.810511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:53.810511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:53.810511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:53.810511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:53.810511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:53.810511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:53.810920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:53.810920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:53.811350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:53.811350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:53.811350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:53.811350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:53.811350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:53.811350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:53.811350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:53.811350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:53.811350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:53.811350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:53.811350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:53.811350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:53.811350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:53.811350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:53.811350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:53.811350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:53.812075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:53.812075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:53.812075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:53.812075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:53.812075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:53.812075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:53.812075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:53.812075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:53.812075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:53.812075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:53.812075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:53.812075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:53.812075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:53.812075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:53.812075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:53.812075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:53.812075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:53.812075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:53.812075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:53.812554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:53.812554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:53.812554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:55.855411 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:55.856801 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:55.856801 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:55.857070 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:55.858203 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:55.858203 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:55.858203 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:55.858203 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:55.858203 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:55.858203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.858777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:55.858777 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:55.858777 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:55.858777 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:55.858777 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:55.858777 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:55.858777 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.858777 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:55.862370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.862370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:55.862370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:55.862370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:55.862370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.862370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:55.862370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:55.862370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.862370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:55.862370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:55.862370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:55.862370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.862703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:55.862703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:55.862703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:55.862703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:55.862703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:55.862703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:55.862703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:55.862703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:55.862703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:55.862703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:55.862703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:55.862703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:55.862703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:55.862940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.862940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:55.862940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.862940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:55.862940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:55.862940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:55.862940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:55.862940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:55.862940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.862940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:55.862940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:55.862940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.862940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:55.862940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:55.862940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.863239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:55.863239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:55.863239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.863239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:39:55.863239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:55.863239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.863239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:55.863239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:55.863239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:55.863239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.863239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:55.863239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:55.863239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:55.863239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:55.863239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:55.863239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:55.863239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:55.863239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:55.863239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:55.863781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:55.863781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:55.863781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:55.863781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:55.863781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.863781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:55.866573 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:55.866803 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:55.866911 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:55.867006 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:39:55.867410 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:55.867601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.868237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:55.868237 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:55.868237 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:55.868237 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:55.868237 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:55.868237 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:55.868237 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.868237 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:55.868870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.868870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:55.868870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:55.868870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:55.868870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.868870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:55.868870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:55.868870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:55.868870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.868870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:55.868870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:55.868870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:55.868870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:55.868870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:55.868870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:55.869238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:55.869238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:55.869238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:55.869238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:55.869238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:55.869238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:55.869238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:55.869238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.869238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:55.869571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.869571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:55.869571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:55.869571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:55.869571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.869571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:55.869571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:55.869571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:39:55.869571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:55.869571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:55.869571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.869571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:55.870186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:55.870186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:55.870186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.870186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:55.870186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:55.870186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:55.870186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:55.870186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:55.870186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:55.870186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:55.870186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:55.870186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:55.870186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:55.870186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:55.870745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:55.870745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:55.870745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.870745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:55.870745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.870745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:55.870745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:55.870745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:55.870745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:55.870745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:55.870745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.870745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:55.870745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:55.870745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:55.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:55.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:55.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:55.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:39:55.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:39:55.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:39:55.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:55.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:55.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:55.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:55.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:55.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:55.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:55.870963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:55.871274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:55.871274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:55.871274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:55.871274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:55.871274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:55.871274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:55.871274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:55.871274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:55.871274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:55.871274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:55.871274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.871274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:55.874634 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:55.875754 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:55.875754 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:55.875754 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:55.876072 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:55.876305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.878798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:55.878798 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:55.878798 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:55.878798 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:55.878798 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:55.878798 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:55.878798 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.878798 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:55.878798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.878798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:39:55.878798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:55.878798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:55.878798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.878798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:55.878798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:55.878798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:39:55.878798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.878798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:39:55.879111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:55.879111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:39:55.879111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:55.879111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:39:55.879111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:39:55.879111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:55.879111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:39:55.879111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:55.879111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:39:55.879111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:39:55.879111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:55.879111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:39:55.879111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.879111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:39:55.879111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.879383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:55.879383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:55.879383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:55.879383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.879383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:55.879383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:55.879383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:55.879383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:55.879383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.879383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:55.879383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:55.879383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:55.879383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.879607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:55.879607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:55.879607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:55.879607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:55.879607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:55.879607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:55.879607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:55.879607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:55.879607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:55.879607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:55.879607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:55.879607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:55.879607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:55.879888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.879888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:55.879888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.879888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:55.879888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:55.879888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:55.879888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:55.879888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:55.879888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.879888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:55.879888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:55.879888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.879888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:55.879888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:55.879888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.880197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:55.880197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:55.880197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.880197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:39:55.880197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:39:55.880197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:39:55.880197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:39:55.880197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.880197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:55.880197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:55.880197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:55.880197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.880197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:55.880197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:55.880197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:55.880197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:55.880197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:55.880197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:55.880197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:55.880506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:55.880506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:55.880506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:55.880506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:55.880506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:55.880506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:55.880506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.880506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:55.892117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:55.893001 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:55.893001 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:55.893001 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:55.893282 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:55.893556 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:55.893795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.894294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:55.894294 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:55.894294 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:55.894294 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:55.894294 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:55.894294 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:55.894294 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.894294 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:55.896104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.896104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:55.896104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:55.896104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:55.896104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.896104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:55.896104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:55.896104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.896104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:55.896104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:55.896104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:55.896104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.896368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:55.896368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:55.896368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:55.896368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:55.896368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:55.896368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:55.896368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:55.896368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:55.896368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:55.896368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:55.896368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:55.896368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:55.896368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:55.896597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.896597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:55.896597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.896597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:55.896597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:55.896597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:55.896597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:55.896597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:55.896597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.896597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:55.896597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:55.896597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.896597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:55.896597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:55.896597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.896852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:55.896852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:55.896852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:55.896852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:39:55.896852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:55.896852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.896852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:55.896852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:55.896852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:55.896852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:55.896852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:55.896852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:55.896852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:55.896852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:55.896852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:55.896852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:55.896852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:55.896852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:55.896852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:55.897150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:55.897150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:55.897150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:55.897150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:55.897150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:55.897150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:39:58.775635 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:39:58.776163 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:39:58.776163 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:39:58.776799 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:39:58.805510 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:39:58.805735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:58.806222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:39:58.806222 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:58.806222 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:39:58.806222 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:39:58.806222 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:58.806222 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:39:58.806222 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:58.806222 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:39:58.806939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:58.806939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:39:58.806939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:58.806939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:39:58.806939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:58.806939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:58.806939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:58.806939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:58.806939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:58.806939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:58.806939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:39:58.806939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:58.807152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:39:58.807152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:58.807152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:39:58.807152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:58.807152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:39:58.807152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:39:58.807152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:58.807152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:39:58.807152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:58.807152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:39:58.807152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:39:58.807152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:58.807152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:39:58.807372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:58.807372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:39:58.807760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:58.807760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:39:58.807760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:39:58.807760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:39:58.807760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:39:58.807760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:39:58.807760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:58.807760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:39:58.807760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:39:58.807760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:58.807760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:39:58.807760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:39:58.807760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:39:58.807760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:39:58.807760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:39:58.807760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:39:58.808041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:39:58.808041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:39:58.808041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:58.808041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:58.808041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:39:58.808041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:39:58.808041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:39:58.808041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:39:58.808041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:39:58.808041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:39:58.808041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:39:58.808041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:39:58.808041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:39:58.808041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:39:58.808041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:39:58.808041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:39:58.808041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:39:58.808041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:39:58.808041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:39:58.808345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:39:58.808345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:39:58.808345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:00.856269 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:00.856722 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:00.856777 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:00.857643 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:00.857868 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:00.857996 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:00.858112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:00.858265 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:00.858388 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:00.858494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.858974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:00.858974 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:00.858974 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:00.858974 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:00.858974 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:00.858974 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:00.858974 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.858974 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:00.859739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.859739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:00.859739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:00.859739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:00.859739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.859739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:00.859739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:00.859739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.859739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:00.859739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:00.859739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:00.859739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.859990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:00.859990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:00.859990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:00.859990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:00.859990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:00.859990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:00.859990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:00.859990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:00.859990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:00.859990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:00.859990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:00.859990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:00.859990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:00.860197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.860197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:00.860639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.860639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:00.860639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:00.860639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:00.860639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:00.860639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:00.860639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.860639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:00.860639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:00.860639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.860639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:00.860639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:00.860639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.860639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:00.860639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:00.860639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.860895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:00.860895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:00.860895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.860895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:00.860895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:00.860895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:00.860895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.860895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:00.860895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:00.860895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:00.860895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:00.860895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:00.860895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:00.860895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:00.860895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:00.860895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:00.860895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:00.860895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:00.860895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:00.861191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:00.861191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.861191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:00.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:00.867245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:00.867360 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:00.867461 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:40:00.870345 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:00.870345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.870487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:00.870487 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:00.870487 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:00.870487 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:00.870487 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:00.870487 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:00.870487 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.870487 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:00.871074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.871074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:00.871074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:00.871074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:00.871074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.871074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:00.871074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:00.871074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:00.871074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.871074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:00.871074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:00.871074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:00.871074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:00.871074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:00.871074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:00.871471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:00.871471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:00.871471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:00.871471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:00.871471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:00.871471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:00.871471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:00.871471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.871471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:00.871881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.871881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:00.871881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:00.871881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:00.871881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.871881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:00.871881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:00.871881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:40:00.871881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:00.871881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:00.871881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.871881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:00.872452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:00.872452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:00.872452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.872452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:00.872452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:00.872452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:00.872452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:00.872452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:00.872452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:00.872452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:00.872452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:00.872452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:00.872452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:00.872452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:00.873044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:00.873044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:00.873044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.873044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:00.873044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.873044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:00.873044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:00.873044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:00.873044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:00.873044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:00.873044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.873044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:00.873044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:00.873044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.873505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:00.873505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:00.873505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.873505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:00.873505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:00.873505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.873505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:00.873505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:00.873505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:40:00.873505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:00.873505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:00.873505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.873505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:00.873505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:00.873505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:00.873505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.873505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:00.873505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:00.873505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:00.873963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:00.873963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:00.873963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:00.873963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:00.873963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:00.873963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:00.873963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:00.873963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:00.873963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:00.873963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:00.873963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.873963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:00.875586 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:00.875836 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:00.875921 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:00.876007 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:00.877090 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:00.877090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.877237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:00.877237 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:00.877237 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:00.877237 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:00.877237 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:00.877237 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:00.877237 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.877237 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:00.877769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.877769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:00.877769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:00.877769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:00.877769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.877769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:00.877769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:00.877769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:00.877769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.877769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:00.877769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:00.877769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:00.877769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:00.877769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:00.877769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:00.878513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:00.878513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:00.878513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:00.878513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:00.878513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:00.878513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:00.878513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:00.878513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.878513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:00.878513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.878513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:00.878513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:00.878513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:00.878513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.879184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:00.879184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:00.879184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:00.879184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:00.879184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.879184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:00.879184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:00.879184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:00.879184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.879184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:00.879184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:00.879184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:00.879184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:00.879514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:00.879514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:00.879514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:00.879514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:00.879514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:00.879514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:00.879514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:00.879514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:00.879514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:00.879514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.879514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:00.879514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.879829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:00.879829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:00.879829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:00.879829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:00.879829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:00.879829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.879829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:00.879829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:00.879829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.879829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:00.879829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:00.879829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.879829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:00.879829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:00.879829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.879829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:00.879829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:00.879829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:00.879829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.880280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:00.892609 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:00.892915 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:00.893019 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:00.893153 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:00.894211 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:00.894211 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:00.894211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.894372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:00.894372 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:00.894372 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:00.894372 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:00.894372 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:00.894372 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:00.894372 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.894372 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:00.895045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.895045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:00.895045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:00.895045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:00.895045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.895045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:00.895045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:00.895045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.895045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:00.895045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:00.895045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:00.895045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.895280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:00.895280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:00.895280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:00.895280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:00.895280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:00.895280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:00.895280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:00.895280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:00.895280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:00.895280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:00.895280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:00.895280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:00.895280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:00.895553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.895553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:00.895867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.895867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:00.895867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:00.895867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:00.895867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:00.895867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:00.895867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.895867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:00.895867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:00.895867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.895867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:00.895867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:00.895867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.895867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:00.895867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:00.895867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:00.896543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:00.896543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:00.896543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.896543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:00.896543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:00.896543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:00.896543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:00.896543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:00.896543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:00.896543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:00.896543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:00.896543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:00.896543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:00.896543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:00.896543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:00.896543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:00.896543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:00.896543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:00.896543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:00.897216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:00.897216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:00.897216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:03.776109 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:03.776684 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:03.776684 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:03.777236 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:03.810937 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:03.811554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:03.811797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:03.811797 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:03.811797 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:03.811797 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:03.811797 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:03.811797 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:03.811797 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:03.811797 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:03.813627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:03.813627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:03.813627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:03.813627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:03.813627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:03.813627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:03.813627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:03.813627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:03.813627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:03.813627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:03.813627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:03.813627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:03.814047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:03.814047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:03.814047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:03.814047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:03.814047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:03.814047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:03.814047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:03.814047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:03.814047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:03.814047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:03.814047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:03.814047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:03.814047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:03.814267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:03.814267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:03.814267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:03.814267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:03.814267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:03.814267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:03.814267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:03.814267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:03.814267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:03.814267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:03.814267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:03.814267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:03.814267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:03.814267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:03.814267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:03.814495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:03.814495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:03.814495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:03.814495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:03.814495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:03.814495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:03.814495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:03.814495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:03.814495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:03.814495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:03.814495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:03.814495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:03.814495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:03.814495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:03.814495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:03.814495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:03.814495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:03.814495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:03.814495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:03.814810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:03.814810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:03.814810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:03.814810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:03.814810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:03.814810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:05.856748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:05.858353 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:05.858353 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:05.858787 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:05.859054 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:05.859318 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:05.860365 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:05.860365 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:05.860365 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:05.860365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:05.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:05.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:05.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:05.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:05.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:05.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.860790 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:05.863044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.863044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:05.863044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:05.863044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:05.863044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.863044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:05.863044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:05.863044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.863044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:05.863044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:05.863044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:05.863044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.863437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:05.863437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:05.863437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:05.863437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:05.863437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:05.863437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:05.863437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:05.863437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:05.863437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:05.863437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:05.863437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:05.863437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:05.863437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:05.863797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.863797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:05.863797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.863797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:05.863797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:05.863797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:05.863797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:05.863797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:05.863797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.863797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:05.863797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:05.863797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.863797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:05.863797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:05.863797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.864200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:05.864200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:05.864200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.864200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:05.864200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:05.864200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.864200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:05.864200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:05.864200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:05.864200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.864200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:05.864200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:05.864200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:05.864200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:05.864200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:05.864200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:05.864200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:05.864200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:05.864200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:05.864726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:05.864726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:05.864726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:05.864726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:05.864726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.864726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:05.867266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:05.867838 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:05.867838 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:05.867838 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:40:05.868194 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:05.868478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.869019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:05.869019 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:05.869019 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:05.869019 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:05.869019 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:05.869019 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:05.869019 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.869019 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:05.871268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.871268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:05.871268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:05.871268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:05.871268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.871268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:05.871268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:05.871268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:05.871268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.871268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:05.871268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:05.871268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:05.871268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:05.871268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:05.871268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:05.871688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:05.871688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:05.871688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:05.871688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:05.871688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:05.871688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:05.871688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:05.871688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.871688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:05.871688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.871688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:05.871688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:05.871688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:05.871688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:05.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:05.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:40:05.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:05.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:05.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:05.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:05.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:05.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:05.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:05.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:05.872362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:05.872362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:05.872362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:05.872362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:05.872362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:05.872362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:05.872362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:05.872362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:05.872362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:05.872362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:05.872362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.872362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:05.873788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.873788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:05.873788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:05.873788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:05.873788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:05.873788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:05.873788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.873788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:05.873788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:05.873788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.873788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:05.873788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:05.873788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.873788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:05.873788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:05.873788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.874650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:05.874650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:05.874650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:40:05.874650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:05.874650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:05.874650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.874650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:05.874650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:05.874650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:05.874650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.874650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:05.874650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:05.874650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:05.874650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:05.874650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:05.874650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:05.874650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:05.874650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:05.875499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:05.875499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:05.875499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:05.875499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:05.875499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:05.875499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.875499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:05.875897 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:05.877262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:05.877262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:05.877262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:05.877262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:05.877262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.877630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:05.877630 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:05.877630 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:05.877630 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:05.877630 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:05.877630 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:05.877630 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.877630 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:05.880375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.880375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:05.880375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:05.880375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:05.880375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.880375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:05.880375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:05.880375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:05.880375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.880375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:05.880375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:05.880375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:05.880375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:05.880375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:05.880375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:05.880723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:05.880723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:05.880723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:05.880723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:05.880723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:05.880723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:05.880723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:05.880723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.880723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:05.880723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.880723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:05.880723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:05.880723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:05.880723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.881084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:05.881084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:05.881084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:05.881084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:05.881084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.881084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:05.881084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:05.881084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:05.881084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.881084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:05.881084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:05.881084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:05.881084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:05.881406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:05.881406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:05.881406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:05.881406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:05.881406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:05.881406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:05.881406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:05.881406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:05.881406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:05.881406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.881406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:05.882462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.882462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:05.882462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:05.882462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:05.882462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:05.882462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:05.882462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.882462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:05.882462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:05.882462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.882462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:05.882462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:05.882462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.882462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:05.882462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:05.882462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.882462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:05.883425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:05.883425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:05.883425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:05.883425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.883425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:05.883425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:05.883425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:05.883425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.883425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:05.883425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:05.883425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:05.883425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:05.883425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:05.883425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:05.883425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:05.883425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:05.883425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:05.883425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:05.883425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:05.884311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:05.884311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:05.884311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.884311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:05.893036 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:05.893943 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:05.893943 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:05.893943 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:05.894350 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:05.894697 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:05.895034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.895660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:05.895660 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:05.895660 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:05.895660 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:05.895660 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:05.895660 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:05.895660 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.895660 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:05.897870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.897870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:05.897870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:05.897870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:05.897870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.897870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:05.897870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:05.897870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.897870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:05.897870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:05.897870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:05.897870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.898204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:05.898204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:05.898204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:05.898204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:05.898204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:05.898204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:05.898204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:05.898204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:05.898204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:05.898204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:05.898204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:05.898204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:05.898204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:05.898529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.898529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:05.898529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.898529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:05.898529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:05.898529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:05.898529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:05.898529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:05.898529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.898529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:05.898529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:05.898529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.898529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:05.898529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:05.898529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.898903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:05.898903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:05.898903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:05.898903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:05.898903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:05.898903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.898903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:05.898903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:05.898903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:05.898903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:05.898903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:05.898903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:05.898903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:05.898903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:05.898903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:05.898903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:05.898903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:05.898903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:05.898903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:05.899372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:05.899372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:05.899372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:05.899372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:05.899372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:05.899372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:08.776462 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:08.778795 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:08.778795 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:08.778795 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:08.811131 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:08.811846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:08.812322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:08.812322 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:08.812322 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:08.812322 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:08.812322 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:08.812322 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:08.812322 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:08.812322 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:08.814122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:08.814122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:08.814122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:08.814122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:08.814122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:08.814122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:08.814122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:08.814122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:08.814122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:08.814122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:08.814122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:08.814122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:08.814443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:08.814443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:08.814443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:08.814443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:08.814443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:08.814443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:08.814443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:08.814443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:08.814443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:08.814443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:08.814443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:08.814443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:08.814443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:08.814656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:08.814656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:08.814656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:08.814656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:08.814656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:08.814656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:08.814656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:08.814656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:08.814656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:08.814656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:08.814656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:08.814656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:08.814656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:08.814656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:08.814656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:08.814912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:08.814912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:08.814912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:08.814912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:08.814912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:08.814912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:08.814912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:08.814912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:08.814912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:08.814912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:08.814912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:08.814912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:08.814912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:08.814912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:08.814912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:08.814912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:08.814912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:08.814912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:08.814912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:08.815285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:08.815285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:08.815285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:08.815285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:08.815285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:08.815285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:10.857784 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:10.859457 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:10.859457 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:10.860094 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:10.860569 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:10.860884 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:10.861143 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:10.861543 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:10.861873 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:10.862163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.863107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:10.863107 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:10.863107 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:10.863107 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:10.863107 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:10.863107 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:10.863107 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.863107 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:10.864907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.864907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:10.864907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:10.864907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:10.864907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.864907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:10.864907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:10.864907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.864907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:10.864907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:10.864907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:10.864907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.865809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:10.865809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:10.865809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:10.865809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:10.865809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:10.865809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:10.865809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:10.865809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:10.865809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:10.865809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:10.865809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:10.865809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:10.865809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:10.866162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.866162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:10.867110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.867110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:10.867110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:10.867110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:10.867110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:10.867110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:10.867110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.867110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:10.867110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:10.867110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.867110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:10.867110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:10.867110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.867110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:10.867110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:10.867110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.867530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:10.867530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:10.867530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.867530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:10.867530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:10.867530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:10.867530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.867530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:10.867530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:10.867530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:10.867530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:10.867530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:10.867530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:10.867530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:10.867530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:10.867530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:10.867530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:10.867530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:10.867530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:10.868030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:10.868030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.868030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:10.868030 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:10.868030 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:10.868152 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:10.868192 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:40:10.868998 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:10.868998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.869107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:10.869107 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:10.869107 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:10.869107 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:10.869107 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:10.869107 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:10.869107 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.869107 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:10.869705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.869705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:10.869705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:10.869705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:10.869705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.869705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:10.869705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:10.869705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:10.869705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.869705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:10.869705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:10.869705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:10.869705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:10.869705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:10.869705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:10.870382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:10.870382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:10.870382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:10.870382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:10.870382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:10.870382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:10.870382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:10.870382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.870382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:10.870382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.870382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:10.870382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:10.870382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:10.870382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.870708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:10.870708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:10.870708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:40:10.870708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:10.870708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:10.870708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.870708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:10.870708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:10.870708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:10.870708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.870708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:10.870708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:10.870708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:10.871104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:10.871104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:10.871104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:10.871104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:10.871104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:10.871104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:10.871104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:10.871104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:10.871104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:10.871104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:10.871104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.871104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:10.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:10.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:10.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:10.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:10.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:10.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:10.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:10.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:10.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:10.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:10.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:10.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.872199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:10.872199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:10.872199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:40:10.872199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:10.872199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:10.872199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.872199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:10.872199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:10.872199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:10.872199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.872199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:10.872199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:10.872199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:10.872199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:10.872199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:10.872199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:10.872199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:10.872199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:10.872680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:10.872680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:10.872680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:10.872680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:10.872680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:10.872680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.872680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:10.876302 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:10.876619 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:10.876763 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:10.876906 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:10.877891 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:10.877891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.878008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:10.878008 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:10.878008 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:10.878008 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:10.878008 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:10.878008 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:10.878008 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.878008 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:10.878745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.878745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:10.878745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:10.878745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:10.878745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.878745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:10.878745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:10.878745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:10.878745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.878745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:10.878745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:10.878745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:10.878745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:10.878745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:10.878745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:10.879106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:10.879106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:10.879106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:10.879106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:10.879106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:10.879106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:10.879106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:10.879106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.879106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:10.879640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.879640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:10.879640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:10.879640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:10.879640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.879640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:10.879640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:10.879640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:10.879640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:10.879640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.879640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:10.879936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:10.879936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:10.879936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.879936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:10.879936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:10.879936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:10.879936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:10.879936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:10.879936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:10.879936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:10.879936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:10.879936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:10.879936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:10.879936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:10.880303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:10.880303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:10.880303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.880303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:10.880707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.880707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:10.880707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:10.880707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:10.880707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:10.880707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:10.880707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.880707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:10.880707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:10.880707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.880707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:10.880707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:10.880707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.880707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:10.880707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:10.880707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.880707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:10.881116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:10.881116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:10.881116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:10.881116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.881116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:10.881116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:10.881116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:10.881116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.881116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:10.881116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:10.881116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:10.881116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:10.881116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:10.881116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:10.881116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:10.881116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:10.881116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:10.881116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:10.881116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:10.881559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:10.881559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:10.881559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.881559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:10.893589 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:10.893908 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:10.894009 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:10.894147 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:10.895112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:10.895112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:10.895112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.895220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:10.895220 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:10.895220 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:10.895220 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:10.895220 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:10.895220 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:10.895220 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.895220 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:10.895842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.895842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:10.895842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:10.895842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:10.895842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.895842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:10.895842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:10.895842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.895842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:10.895842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:10.895842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:10.895842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.896110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:10.896110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:10.896110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:10.896110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:10.896110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:10.896110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:10.896110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:10.896110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:10.896110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:10.896110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:10.896110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:10.896110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:10.896110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:10.896370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.896370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:10.896540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.896540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:10.896540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:10.896540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:10.896540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:10.896540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:10.896540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.896540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:10.896540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:10.896540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.896540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:10.896540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:10.896540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.896540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:10.896540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:10.896540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:10.896832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:10.896832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:10.896832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.896832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:10.896832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:10.896832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:10.896832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:10.896832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:10.896832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:10.896832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:10.896832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:10.896832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:10.896832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:10.896832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:10.896832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:10.896832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:10.896832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:10.896832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:10.896832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:10.897162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:10.897162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:10.897162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:13.777426 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:13.777875 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:13.777875 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:13.778370 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:13.807891 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:13.807891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:13.808091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:13.808091 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:13.808091 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:13.808091 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:13.808091 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:13.808091 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:13.808091 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:13.808091 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:13.808820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:13.808820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:13.808820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:13.808820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:13.808820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:13.808820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:13.808820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:13.808820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:13.808820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:13.808820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:13.808820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:13.808820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:13.809073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:13.809073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:13.809073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:13.809073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:13.809073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:13.809073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:13.809073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:13.809073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:13.809073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:13.809073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:13.809073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:13.809073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:13.809073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:13.809328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:13.809328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:13.809592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:13.809592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:13.809592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:13.809592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:13.809592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:13.809592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:13.809592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:13.809592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:13.809592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:13.809592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:13.809592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:13.809592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:13.809592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:13.809592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:13.809592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:13.809592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:13.809901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:13.809901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:13.809901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:13.809901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:13.809901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:13.809901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:13.809901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:13.809901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:13.809901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:13.809901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:13.809901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:13.809901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:13.809901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:13.809901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:13.809901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:13.809901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:13.809901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:13.809901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:13.809901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:13.810209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:13.810209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:13.810209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:15.858718 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:15.859088 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:15.859217 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:15.859994 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:15.860145 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:15.860267 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:15.860407 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:15.860516 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:15.860650 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:15.860755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.861183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:15.861183 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:15.861183 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:15.861183 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:15.861183 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:15.861183 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:15.861183 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.861183 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:15.861838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.861838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:15.861838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:15.861838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:15.861838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.861838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:15.861838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:15.861838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.861838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:15.861838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:15.861838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:15.861838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.862161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:15.862161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:15.862161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:15.862161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:15.862161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:15.862161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:15.862161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:15.862161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:15.862161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:15.862161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:15.862161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:15.862161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:15.862161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:15.862419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.862419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:15.862545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.862545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:15.862545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:15.862545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:15.862545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:15.862545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:15.862545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.862545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:15.862545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:15.862545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.862545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:15.862545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:15.862545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.862545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:15.862545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:15.862545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.862885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:15.862885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:15.862885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.862885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:15.862885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:15.862885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:15.862885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.862885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:15.862885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:15.862885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:15.862885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:15.862885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:15.862885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:15.862885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:15.862885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:15.862885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:15.862885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:15.862885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:15.862885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:15.863277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:15.863277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.863277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:15.868070 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:15.868304 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:15.868391 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:15.868488 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:40:15.868723 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:15.868855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.869187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:15.869187 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:15.869187 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:15.869187 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:15.869187 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:15.869187 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:15.869187 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.869187 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:15.869729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.869729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:15.869729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:15.869729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:15.869729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.869729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:15.869729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:15.869729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:15.869729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.869729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:15.869729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:15.869729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:15.869729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:15.869729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:15.869729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:15.870015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:15.870015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:15.870015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:15.870015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:15.870015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:15.870015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:15.870015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:15.870015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.870015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:15.870266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.870266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:15.870266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:15.870266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:15.870266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.870266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:15.870266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:15.870266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:40:15.870266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:15.870266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:15.870266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.870266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:15.870529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:15.870529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:15.870529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.870529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:15.870529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:15.870529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:15.870529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:15.870529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:15.870529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:15.870529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:15.870529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:15.870529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:15.870529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:15.870529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:15.870760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:15.870760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:15.870760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.870760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:15.870932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.870932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:15.870932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:15.870932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:15.870932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:15.870932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:15.870932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.870932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:15.870932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:15.870932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.870932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:15.870932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:15.870932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.870932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:15.870932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:15.870932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:15.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:15.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:40:15.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:15.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:15.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:15.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:15.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:15.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:15.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:15.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:15.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:15.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:15.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:15.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:15.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:15.871555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:15.871555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:15.871555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:15.871555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:15.871555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:15.871555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.871555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:15.876646 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:15.876853 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:15.876940 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:15.877036 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:15.877264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:15.877438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.877794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:15.877794 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:15.877794 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:15.877794 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:15.877794 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:15.877794 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:15.877794 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.877794 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:15.878262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.878262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:15.878262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:15.878262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:15.878262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.878262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:15.878262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:15.878262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:15.878262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.878262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:15.878262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:15.878262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:15.878262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:15.878262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:15.878262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:15.878566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:15.878566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:15.878566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:15.878566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:15.878566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:15.878566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:15.878566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:15.878566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.878566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:15.878827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.878827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:15.878827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:15.878827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:15.878827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.878827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:15.878827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:15.878827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:15.878827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:15.878827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.878827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:15.879048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:15.879048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:15.879048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.879048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:15.879048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:15.879048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:15.879048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:15.879048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:15.879048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:15.879048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:15.879048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:15.879048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:15.879048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:15.879048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:15.879267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:15.879267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:15.879267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.879267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:15.879581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.879581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:15.879581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:15.879581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:15.879581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:15.879581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:15.879581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.879581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:15.879581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:15.879581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.879581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:15.879581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:15.879581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.879581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:15.879581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:15.879581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.879581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:15.879883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:15.879883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:15.879883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:15.879883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.879883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:15.879883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:15.879883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:15.879883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.879883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:15.879883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:15.879883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:15.879883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:15.879883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:15.879883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:15.879883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:15.879883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:15.879883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:15.879883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:15.879883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:15.880174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:15.880174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:15.880174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.880174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:15.894006 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:15.894225 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:15.894331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:15.894466 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:15.894835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:15.895041 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:15.895215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.895730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:15.895730 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:15.895730 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:15.895730 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:15.895730 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:15.895730 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:15.895730 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.895730 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:15.896373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.896373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:15.896373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:15.896373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:15.896373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.896373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:15.896373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:15.896373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.896373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:15.896373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:15.896373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:15.896373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.896641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:15.896641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:15.896641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:15.896641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:15.896641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:15.896641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:15.896641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:15.896641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:15.896641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:15.896641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:15.896641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:15.896641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:15.896641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:15.896883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.896883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:15.897062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.897062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:15.897062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:15.897062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:15.897062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:15.897062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:15.897062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.897062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:15.897062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:15.897062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.897062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:15.897062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:15.897062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.897062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:15.897062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:15.897062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:15.897360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:15.897360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:15.897360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.897360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:15.897360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:15.897360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:15.897360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:15.897360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:15.897360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:15.897360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:15.897360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:15.897360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:15.897360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:15.897360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:15.897360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:15.897360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:15.897360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:15.897360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:15.897360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:15.897726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:15.897726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:15.897726 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:18.779488 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:18.779978 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:18.779978 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:18.780783 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:18.813264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:18.813460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:18.813885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:18.813885 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:18.813885 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:18.813885 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:18.813885 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:18.813885 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:18.813885 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:18.813885 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:18.814547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:18.814547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:18.814547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:18.814547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:18.814547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:18.814547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:18.814547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:18.814547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:18.814547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:18.814547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:18.814547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:18.814547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:18.814804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:18.814804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:18.814804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:18.814804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:18.814804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:18.814804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:18.814804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:18.814804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:18.814804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:18.814804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:18.814804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:18.814804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:18.814804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:18.815039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:18.815039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:18.815275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:18.815275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:18.815275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:18.815275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:18.815275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:18.815275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:18.815275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:18.815275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:18.815275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:18.815275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:18.815275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:18.815275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:18.815275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:18.815275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:18.815275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:18.815275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:18.815570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:18.815570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:18.815570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:18.815570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:18.815570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:18.815570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:18.815570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:18.815570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:18.815570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:18.815570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:18.815570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:18.815570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:18.815570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:18.815570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:18.815570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:18.815570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:18.815570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:18.815570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:18.815570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:18.815906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:18.815906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:18.815906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:20.859144 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:20.859728 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:20.859728 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:20.860393 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:20.860591 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:20.860740 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:20.860859 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:20.860985 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:20.861114 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:20.861214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.861709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:20.861709 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:20.861709 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:20.861709 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:20.861709 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:20.861709 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:20.861709 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.861709 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:20.862419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.862419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:20.862419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:20.862419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:20.862419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.862419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:20.862419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:20.862419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.862419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:20.862419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:20.862419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:20.862419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.862740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:20.862740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:20.862740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:20.862740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:20.862740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:20.862740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:20.862740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:20.862740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:20.862740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:20.862740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:20.862740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:20.862740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:20.862740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:20.862946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.862946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:20.863124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.863124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:20.863124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:20.863124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:20.863124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:20.863124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:20.863124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.863124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:20.863124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:20.863124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.863124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:20.863124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:20.863124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.863124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:20.863124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:20.863124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.863410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:20.863410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:20.863410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.863410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:20.863410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:20.863410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:20.863410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.863410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:20.863410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:20.863410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:20.863410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:20.863410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:20.863410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:20.863410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:20.863410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:20.863410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:20.863410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:20.863410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:20.863410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:20.863829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:20.863829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.863829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:20.868430 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:20.868653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:20.868745 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:20.868835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:40:20.869169 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:20.869351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.869764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:20.869764 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:20.869764 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:20.869764 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:20.869764 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:20.869764 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:20.869764 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.869764 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:20.870319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.870319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:20.870319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:20.870319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:20.870319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.870319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:20.870319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:20.870319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:20.870319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.870319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:20.870319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:20.870319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:20.870319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:20.870319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:20.870319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:20.870652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:20.870652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:20.870652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:20.870652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:20.870652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:20.870652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:20.870652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:20.870652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.870652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:20.870879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.870879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:20.870879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:20.870879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:20.870879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.870879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:20.870879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:20.870879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:40:20.870879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:20.870879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:20.870879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.870879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:20.871100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:20.871100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:20.871100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.871100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:20.871100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:20.871100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:20.871100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:20.871100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:20.871100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:20.871100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:20.871100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:20.871100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:20.871100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:20.871100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:20.871372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:20.871372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:20.871372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.871372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:20.871548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.871548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:20.871548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:20.871548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:20.871548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:20.871548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:20.871548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.871548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:20.871548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:20.871548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.871548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:20.871548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:20.871548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.871548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:20.871548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:20.871548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.871835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:20.871835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:20.871835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:40:20.871835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:20.871835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:20.871835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.871835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:20.871835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:20.871835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:20.871835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.871835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:20.871835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:20.871835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:20.871835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:20.871835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:20.871835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:20.871835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:20.871835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:20.872124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:20.872124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:20.872124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:20.872124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:20.872124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:20.872124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.872124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:20.876987 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:20.877193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:20.877301 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:20.877397 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:20.877818 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:20.878033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.878552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:20.878552 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:20.878552 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:20.878552 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:20.878552 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:20.878552 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:20.878552 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.878552 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:20.879128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.879128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:20.879128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:20.879128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:20.879128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.879128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:20.879128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:20.879128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:20.879128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.879128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:20.879128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:20.879128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:20.879128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:20.879128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:20.879128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:20.879431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:20.879431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:20.879431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:20.879431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:20.879431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:20.879431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:20.879431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:20.879431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.879431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:20.879809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.879809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:20.879809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:20.879809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:20.879809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.879809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:20.879809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:20.879809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:20.879809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:20.879809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.879809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:20.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:20.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:20.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:20.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:20.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:20.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:20.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:20.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:20.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:20.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:20.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:20.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:20.880028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:20.880267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:20.880267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:20.880267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.880267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:20.880561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.880561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:20.880561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:20.880561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:20.880561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:20.880561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:20.880561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.880561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:20.880561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:20.880561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.880561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:20.880561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:20.880561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.880561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:20.880561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:20.880561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.880561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:20.880858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:20.880858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:20.880858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:20.880858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.880858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:20.880858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:20.880858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:20.880858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.880858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:20.880858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:20.880858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:20.880858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:20.880858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:20.880858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:20.880858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:20.880858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:20.880858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:20.880858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:20.880858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:20.881231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:20.881231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:20.881231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.881231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:20.894418 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:20.894851 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:20.895040 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:20.895262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:20.895640 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:20.895921 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:20.896166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.897484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:20.897484 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:20.897484 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:20.897484 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:20.897484 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:20.897484 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:20.897484 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.897484 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:20.899659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.899659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:20.899659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:20.899659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:20.899659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.899659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:20.899659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:20.899659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.899659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:20.899659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:20.899659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:20.899659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.899931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:20.899931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:20.899931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:20.899931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:20.899931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:20.899931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:20.899931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:20.899931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:20.899931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:20.899931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:20.899931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:20.899931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:20.899931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:20.900158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.900158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:20.900353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.900353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:20.900353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:20.900353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:20.900353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:20.900353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:20.900353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.900353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:20.900353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:20.900353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.900353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:20.900353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:20.900353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.900353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:20.900353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:20.900353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:20.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:20.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:20.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:20.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:20.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:20.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:20.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:20.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:20.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:20.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:20.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:20.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:20.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:20.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:20.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:20.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:20.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:20.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:20.901037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:20.901037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:20.901037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:23.779909 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:23.780379 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:23.780489 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:23.781305 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:23.811545 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:23.811806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:23.812347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:23.812347 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:23.812347 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:23.812347 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:23.812347 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:23.812347 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:23.812347 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:23.812347 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:23.813056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:23.813056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:23.813056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:23.813056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:23.813056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:23.813056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:23.813056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:23.813056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:23.813056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:23.813056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:23.813056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:23.813056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:23.813444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:23.813444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:23.813444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:23.813444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:23.813444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:23.813444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:23.813444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:23.813444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:23.813444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:23.813444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:23.813444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:23.813444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:23.813444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:23.813799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:23.813799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:23.813799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:23.813799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:23.813799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:23.813799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:23.813799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:23.813799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:23.813799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:23.813799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:23.813799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:23.813799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:23.813799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:23.813799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:23.813799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:23.814163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:23.814163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:23.814163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:23.814163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:23.814163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:23.814163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:23.814163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:23.814163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:23.814163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:23.814163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:23.814163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:23.814163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:23.814163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:23.814163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:23.814163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:23.814163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:23.814163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:23.814163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:23.814163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:23.814548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:23.814548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:23.814548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:23.814548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:23.814548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:23.814548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:25.859589 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:25.860097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:25.860128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:25.860955 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:25.861215 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:25.861365 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:25.861495 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:25.861662 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:25.861783 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3410570e-9501-4758-9f77-dc544595b890, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:25.861862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.862322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:25.862322 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:25.862322 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:25.862322 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:25.862322 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:25.862322 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:25.862322 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.862322 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:25.863145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.863145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:25.863145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:25.863145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:25.863145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.863145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:25.863145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:25.863145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.863145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:25.863145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:25.863145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:25.863145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.863584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:25.863584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:25.863584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:25.863584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:25.863584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:25.863584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:25.863584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:25.863584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:25.863584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:25.863584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:25.863584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:25.863584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:25.863584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:25.864086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.864086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:25.864301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.864301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:25.864301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:25.864301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:25.864301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:25.864301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:25.864301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.864301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:25.864301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:25.864301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.864301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:25.864301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:25.864301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.864301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:25.864301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:25.864301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.864706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:25.864706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:25.864706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.864706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:25.864706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:25.864706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:25.864706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.864706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:25.864706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:25.864706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:25.864706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:25.864706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:25.864706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:25.864706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:25.864706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:25.864706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:25.864706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:25.864706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:25.864706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:25.865057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:25.865057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.865057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:25.868855 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:25.869162 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:25.869322 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:25.869468 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:40:25.869824 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:25.870026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.870534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:25.870534 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:25.870534 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:25.870534 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:25.870534 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:25.870534 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:25.870534 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.870534 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:25.871198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.871198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:25.871198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:25.871198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:25.871198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.871198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:25.871198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:25.871198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:25.871198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.871198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:25.871198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:25.871198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:25.871198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:25.871198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:25.871198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:25.871968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:25.871968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:25.871968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:25.871968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:25.871968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:25.871968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:25.871968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:25.871968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.871968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:25.872269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.872269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:25.872269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:25.872269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:25.872269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.872269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:25.872269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:25.872269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:40:25.872269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:25.872269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:25.872269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.872269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:25.872578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:25.872578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:25.872578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.872578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:25.872578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:25.872578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:25.872578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:25.872578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:25.872578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:25.872578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:25.872578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:25.872578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:25.872578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:25.872578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:25.872957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:25.872957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:25.872957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.872957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:25.873349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.873349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:25.873349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:25.873349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:25.873349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:25.873349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:25.873349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.873349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:25.873349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:25.873349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.873349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:25.873349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:25.873349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.873349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:25.873349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:25.873349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.874243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:25.874243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:25.874243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:40:25.874243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:25.874243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:25.874243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.874243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:25.874243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:25.874243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:25.874243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.874243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:25.874243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:25.874243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:25.874243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:25.874243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:25.874243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:25.874243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:25.874243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:25.875017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:25.875017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:25.875017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:25.875017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:25.875017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:25.875017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.875017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:25.877447 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:25.877770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:25.877901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:25.878076 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:25.878402 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:25.878652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.879122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:25.879122 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:25.879122 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:25.879122 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:25.879122 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:25.879122 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:25.879122 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.879122 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:25.879795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.879795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:25.879795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:25.879795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:25.879795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.879795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:25.879795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:25.879795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:25.879795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.879795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:25.879795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:25.879795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:25.879795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:25.879795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:25.879795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:25.880242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:25.880242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:25.880242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:25.880242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:25.880242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:25.880242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:25.880242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:25.880242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.880242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:25.880710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.880710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:25.880710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:25.880710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:25.880710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.880710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:25.880710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:25.880710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:25.880710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:25.880710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.880710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:25.881060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:25.881060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:25.881060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.881060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:25.881060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:25.881060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:25.881060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:25.881060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:25.881060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:25.881060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:25.881060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:25.881060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:25.881060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:25.881060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:25.881489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:25.881489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:25.881489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.881489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:25.881842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.881842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:25.881842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:25.881842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:25.881842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:25.881842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:25.881842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.881842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:25.881842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:25.881842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.881842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:25.881842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:25.881842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.881842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:25.881842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:25.881842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.881842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:25.882227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:25.882227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:25.882227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:25.882227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.882227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:25.882227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:25.882227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:25.882227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.882227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:25.882227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:25.882227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:25.882227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:25.882227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:25.882227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:25.882227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:25.882227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:25.882227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:25.882227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:25.882227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:25.882559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:25.882559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:25.882559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.882559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:25.896317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:25.896317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:25.896317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:25.896317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:25.896317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:25.896317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:25.896552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.897031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:25.897031 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:25.897031 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:25.897031 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:25.897031 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:25.897031 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:25.897031 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.897031 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:25.898086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.898086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:25.898086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:25.898086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:25.898086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.898086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:25.898086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:25.898086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.898086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:25.898086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:25.898086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:25.898086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.898518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:25.898518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:25.898518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:25.898518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:25.898518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:25.898518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:25.898518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:25.898518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:25.898518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:25.898518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:25.898518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:25.898518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:25.898518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:25.898925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.898925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:25.899242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.899242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:25.899242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:25.899242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:25.899242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:25.899242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:25.899242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.899242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:25.899242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:25.899242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.899242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:25.899242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:25.899242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.899242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:25.899242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:25.899242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:25.899591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:25.899591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:25.899591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.899591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:25.899591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:25.899591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:25.899591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:25.899591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:25.899591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:25.899591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:25.899591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:25.899591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:25.899591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:25.899591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:25.899591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:25.899591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:25.899591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:25.899591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:25.899591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:25.899963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:25.899963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:25.899963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:28.780356 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:28.780876 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:28.780876 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:28.781414 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:28.810791 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:28.810983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:28.811415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:28.811415 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:28.811415 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:28.811415 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:28.811415 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:28.811415 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:28.811415 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:28.811415 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:28.812120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:28.812120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:28.812120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:28.812120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:28.812120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:28.812120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:28.812120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:28.812120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:28.812120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:28.812120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:28.812120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:28.812120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:28.812356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:28.812356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:28.812356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:28.812356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:28.812356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:28.812356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:28.812356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:28.812356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:28.812356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:28.812356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:28.812356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:28.812356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:28.812356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:28.812563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:28.812563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:28.812886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:28.812886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:28.812886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:28.812886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:28.812886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:28.812886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:28.812886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:28.812886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:28.812886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:28.812886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:28.812886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:28.812886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:28.812886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:28.812886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:28.812886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:28.812886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:28.813186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:28.813186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:28.813186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:28.813186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:28.813186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:28.813186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:28.813186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:28.813186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:28.813186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:28.813186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:28.813186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:28.813186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:28.813186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:28.813186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:28.813186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:28.813186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:28.813186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:28.813186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:28.813186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:28.813519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:28.813519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:28.813519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:30.860043 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:30.860500 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:30.860831 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:30.861812 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:30.861989 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:30.862100 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:30.862207 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:30.862379 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:30.862462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.863290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:30.863290 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:30.863290 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:30.863290 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:30.863290 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:30.863290 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:30.863290 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.863290 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:30.864007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.864007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:30.864007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:30.864007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:30.864007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.864007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:30.864007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:30.864007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.864007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:30.864007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:30.864007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:30.864007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:30.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:30.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:30.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:30.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:30.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:30.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:30.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:30.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:30.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:30.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:30.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:30.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:30.864499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.864499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:30.865150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.865150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:30.865150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:30.865150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:30.865150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:30.865150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:30.865150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.865150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:30.865150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:30.865150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.865150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:30.865150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:30.865150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.865150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:30.865150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:30.865150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:30.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:30.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:30.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:30.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:30.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:30.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:30.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:30.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:30.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:30.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:30.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:30.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:30.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:30.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:30.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:30.865451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:30.865774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:30.865774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.865774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:30.869314 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:30.869658 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:30.869784 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:30.869935 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:40:30.870245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:30.870438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.870871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:30.870871 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:30.870871 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:30.870871 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:30.870871 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:30.870871 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:30.870871 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.870871 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:30.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:30.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:30.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:30.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:30.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:30.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:30.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:30.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:30.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:30.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:30.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:30.871775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:30.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:30.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:30.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:30.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:30.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:30.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:30.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:30.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.872045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:30.872717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.872717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:30.872717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:30.872717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:30.872717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.872717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:30.872717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:30.872717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:40:30.872717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:30.872717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:30.872717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.872717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:30.872961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:30.872961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:30.872961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.872961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:30.872961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:30.872961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:30.872961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:30.872961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:30.872961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:30.872961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:30.872961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:30.872961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:30.872961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:30.872961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:30.873189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:30.873189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:30.873189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.873189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:30.873881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.873881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:30.873881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:30.873881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:30.873881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:30.873881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:30.873881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.873881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:30.873881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:30.873881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.873881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:30.873881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:30.873881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.873881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:30.873881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:30.873881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.874152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:30.874152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:30.874152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:40:30.874152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:30.874152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:30.874152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.874152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:30.874152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:30.874152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:30.874152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.874152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:30.874152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:30.874152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:30.874152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:30.874152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:30.874152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:30.874152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:30.874152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:30.874478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:30.874478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:30.874478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:30.874478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:30.874478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:30.874478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.874478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:30.877855 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:30.878130 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:30.878297 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:30.878457 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:30.878759 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:30.878914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.879302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:30.879302 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:30.879302 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:30.879302 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:30.879302 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:30.879302 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:30.879302 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.879302 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:30.880155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.880155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:30.880155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:30.880155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:30.880155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.880155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:30.880155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:30.880155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:30.880155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.880155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:30.880155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:30.880155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:30.880155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:30.880155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:30.880155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:30.880429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:30.880429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:30.880429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:30.880429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:30.880429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:30.880429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:30.880429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:30.880429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.880429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:30.881174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.881174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:30.881174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:30.881174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:30.881174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.881174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:30.881174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:30.881174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:30.881174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:30.881174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.881174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:30.881396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:30.881396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:30.881396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.881396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:30.881396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:30.881396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:30.881396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:30.881396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:30.881396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:30.881396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:30.881396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:30.881396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:30.881396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:30.881396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:30.881626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:30.881626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:30.881626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.881626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:30.882401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.882401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:30.882401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:30.882401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:30.882401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:30.882401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:30.882401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.882401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:30.882401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:30.882401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.882401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:30.882401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:30.882401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.882401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:30.882401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:30.882401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.882401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:30.882699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:30.882699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:30.882699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:30.882699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.882699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:30.882699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:30.882699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:30.882699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.882699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:30.882699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:30.882699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:30.882699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:30.882699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:30.882699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:30.882699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:30.882699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:30.882699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:30.882699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:30.882699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:30.883120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:30.883120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:30.883120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.883120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:30.895301 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:30.895747 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:30.895887 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:30.896075 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:30.896367 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:30.896517 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:30.896673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.897095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:30.897095 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:30.897095 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:30.897095 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:30.897095 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:30.897095 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:30.897095 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.897095 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:30.898199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.898199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:30.898199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:30.898199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:30.898199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.898199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:30.898199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:30.898199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.898199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:30.898199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:30.898199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:30.898199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.898503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:30.898503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:30.898503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:30.898503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:30.898503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:30.898503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:30.898503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:30.898503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:30.898503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:30.898503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:30.898503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:30.898503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:30.898503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:30.898806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.898806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:30.899379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.899379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:30.899379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:30.899379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:30.899379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:30.899379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:30.899379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.899379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:30.899379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:30.899379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.899379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:30.899379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:30.899379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.899379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:30.899379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:30.899379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:30.899759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:30.899759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:30.899759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.899759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:30.899759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:30.899759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:30.899759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:30.899759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:30.899759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:30.899759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:30.899759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:30.899759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:30.899759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:30.899759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:30.899759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:30.899759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:30.899759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:30.899759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:30.899759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:30.900304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:30.900304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:30.900304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:33.780793 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:33.781576 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:33.781678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:33.782458 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:33.811984 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:33.811984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:33.812273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:33.812273 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:33.812273 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:33.812273 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:33.812273 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:33.812273 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:33.812273 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:33.812273 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:33.812971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:33.812971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:33.812971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:33.812971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:33.812971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:33.812971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:33.812971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:33.812971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:33.812971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:33.812971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:33.812971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:33.812971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:33.813165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:33.813165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:33.813165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:33.813165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:33.813165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:33.813165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:33.813165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:33.813165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:33.813165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:33.813165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:33.813165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:33.813165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:33.813165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:33.813406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:33.813406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:33.813713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:33.813713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:33.813713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:33.813713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:33.813713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:33.813713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:33.813713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:33.813713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:33.813713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:33.813713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:33.813713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:33.813713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:33.813713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:33.813713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:33.813713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:33.813713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:33.813991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:33.813991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:33.813991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:33.813991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:33.813991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:33.813991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:33.813991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:33.813991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:33.813991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:33.813991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:33.813991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:33.813991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:33.813991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:33.813991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:33.813991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:33.813991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:33.813991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:33.813991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:33.813991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:33.814406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:33.814406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:33.814406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:35.860481 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:35.860987 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:35.860987 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:35.862757 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:35.862757 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-86835811-cac9-4ebe-8de1-f0d8cb25ff86, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:35.862757 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:35.862757 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:35.862757 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ecc30130-1f89-40c8-b44b-2b5da362877a, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:35.862757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.863076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:35.863076 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:35.863076 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:35.863076 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:35.863076 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:35.863076 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:35.863076 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.863076 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:35.863914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.863914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:35.863914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:35.863914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:35.863914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.863914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:35.863914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:35.863914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.863914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:35.863914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:35.863914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:35.863914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.864435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:35.864435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:35.864435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:35.864435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:35.864435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:35.864435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:35.864435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:35.864435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:35.864435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:35.864435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:35.864435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:35.864435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:35.864435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:35.864906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.864906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:35.864990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.864990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:35.864990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:35.864990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:35.864990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:35.864990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:35.864990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.864990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:35.864990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:35.864990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.864990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:35.864990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:35.864990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.864990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:35.864990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:35.864990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:35.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:35.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:35.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:35.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:35.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:35.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:35.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:35.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:35.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:35.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:35.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:35.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:35.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:35.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:35.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:35.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:35.866320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:35.866320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.866320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:35.869653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:35.869904 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:35.870014 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:35.870160 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:40:35.870844 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:35.870844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.870998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:35.870998 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:35.870998 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:35.870998 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:35.870998 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:35.870998 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:35.870998 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.870998 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:35.871728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.871728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:35.871728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:35.871728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:35.871728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.871728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:35.871728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:35.871728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:35.871728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.871728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:35.871728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:35.871728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:35.871728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:35.871728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:35.871728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:35.872273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:35.872273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:35.872273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:35.872273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:35.872273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:35.872273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:35.872273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:35.872273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.872273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:35.872610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.872610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:35.872610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:35.872610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:35.872610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.872610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:35.872610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:35.872610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:40:35.872610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:35.872610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:35.872610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.872610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:35.873009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:35.873009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:35.873009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.873009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:35.873009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:35.873009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:35.873009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:35.873009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:35.873009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:35.873009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:35.873009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:35.873009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:35.873009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:35.873009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:35.873485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:35.873485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:35.873485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.873485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:35.873714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.873714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:35.873714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:35.873714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:35.873714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:35.873714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:35.873714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.873714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:35.873714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:35.873714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.873714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:35.873714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:35.873714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.873714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:35.873714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:35.873714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:35.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:35.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:40:35.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:35.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:35.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:35.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:35.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:35.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:35.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:35.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:35.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:35.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:35.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:35.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:35.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:35.874853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:35.874853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:35.874853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:35.874853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:35.874853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:35.874853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.874853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:35.878162 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:35.878433 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:35.878557 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:35.878725 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:35.879436 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:35.879436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.879587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:35.879587 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:35.879587 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:35.879587 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:35.879587 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:35.879587 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:35.879587 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.879587 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:35.880379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.880379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:35.880379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:35.880379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:35.880379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.880379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:35.880379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:35.880379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:35.880379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.880379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:35.880379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:35.880379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:35.880379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:35.880379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:35.880379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:35.880630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:35.880630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:35.880630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:35.880630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:35.880630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:35.880630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:35.880630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:35.880630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.880630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:35.881260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.881260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:35.881260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:35.881260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:35.881260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.881260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:35.881260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:35.881260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:35.881260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:35.881260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.881260 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:35.881471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:35.881471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:35.881471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.881471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:35.881471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:35.881471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:35.881471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:35.881471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:35.881471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:35.881471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:35.881471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:35.881471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:35.881471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:35.881471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:35.881693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:35.881693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:35.881693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.881693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:35.882439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.882439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:35.882439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:35.882439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:35.882439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:35.882439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:35.882439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.882439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:35.882439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:35.882439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.882439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:35.882439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:35.882439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.882439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:35.882439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:35.882439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.882439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:35.882738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:35.882738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:35.882738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:35.882738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.882738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:35.882738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:35.882738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:35.882738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.882738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:35.882738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:35.882738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:35.882738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:35.882738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:35.882738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:35.882738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:35.882738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:35.882738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:35.882738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:35.882738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:35.883068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:35.883068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:35.883068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.883068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:35.895721 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:35.896093 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:35.896226 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:35.896445 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:35.897636 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:35.897636 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:35.897636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.897805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:35.897805 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:35.897805 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:35.897805 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:35.897805 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:35.897805 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:35.897805 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.897805 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:35.898796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.898796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:35.898796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:35.898796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:35.898796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.898796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:35.898796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:35.898796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.898796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:35.898796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:35.898796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:35.898796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.899125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:35.899125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:35.899125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:35.899125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:35.899125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:35.899125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:35.899125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:35.899125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:35.899125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:35.899125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:35.899125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:35.899125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:35.899125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:35.899349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.899349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:35.900012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.900012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:35.900012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:35.900012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:35.900012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:35.900012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:35.900012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.900012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:35.900012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:35.900012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.900012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:35.900012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:35.900012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.900012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:35.900012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:35.900012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:35.900335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:35.900335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:35.900335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.900335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:35.900335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:35.900335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:35.900335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:35.900335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:35.900335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:35.900335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:35.900335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:35.900335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:35.900335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:35.900335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:35.900335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:35.900335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:35.900335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:35.900335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:35.900335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:35.900667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:35.900667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:35.900667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:38.781710 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:38.782104 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:38.782146 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:38.782868 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:38.821561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:38.821752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:38.822183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:38.822183 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:38.822183 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:38.822183 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:38.822183 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:38.822183 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:38.822183 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:38.822183 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:38.822973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:38.822973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:38.822973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:38.822973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:38.822973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:38.822973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:38.822973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:38.822973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:38.822973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:38.822973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:38.822973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:38.822973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:38.823375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:38.823375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:38.823375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:38.823375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:38.823375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:38.823375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:38.823375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:38.823375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:38.823375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:38.823375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:38.823375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:38.823375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:38.823375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:38.823778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:38.823778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:38.823778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:38.823778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:38.823778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:38.823778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:38.823778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:38.823778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:38.823778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:38.823778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:38.823778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:38.823778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:38.823778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:38.823778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:38.823778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:38.824040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:38.824040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:38.824040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:38.824040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:38.824040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:38.824040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:38.824040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:38.824040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:38.824040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:38.824040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:38.824040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:38.824040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:38.824040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:38.824040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:38.824040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:38.824040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:38.824040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:38.824040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:38.824040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:38.824397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:38.824397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:38.824397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:38.824397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:38.824397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:38.824397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:40.860956 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:40.861471 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:40.861471 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:40.862081 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:40.862245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:40.862411 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:40.862548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.863139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:40.863139 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:40.863139 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:40.863139 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:40.863139 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:40.863139 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:40.863139 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.863139 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:40.863937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.863937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:40.863937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:40.863937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:40.863937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.863937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:40.863937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:40.863937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.863937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:40.863937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:40.863937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:40.863937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.864332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:40.864332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:40.864332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:40.864332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:40.864332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:40.864332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:40.864332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:40.864332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:40.864332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:40.864332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:40.864332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:40.864332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:40.864332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:40.864755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.864755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:40.865043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.865043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:40.865043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:40.865043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:40.865043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:40.865043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:40.865043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.865043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:40.865043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:40.865043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.865043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:40.865043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:40.865043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.865043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:40.865043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:40.865043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.865430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:40.865430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:40.865430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.865430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:40.865430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:40.865430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:40.865430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.865430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:40.865430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:40.865430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:40.865430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:40.865430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:40.865430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:40.865430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:40.865430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:40.865430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:40.865430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:40.865430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:40.865430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:40.865800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:40.865800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.865800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:40.870063 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:40.870742 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:40.870742 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:40.870742 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:40:40.871065 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:40.871217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.871742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:40.871742 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:40.871742 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:40.871742 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:40.871742 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:40.871742 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:40.871742 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.871742 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:40.872538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.872538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:40.872538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:40.872538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:40.872538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.872538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:40.872538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:40.872538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:40.872538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.872538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:40.872538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:40.872538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:40.872538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:40.872538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:40.872538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:40.872947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:40.872947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:40.872947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:40.872947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:40.872947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:40.872947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:40.872947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:40.872947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.872947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:40.873348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.873348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:40.873348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:40.873348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:40.873348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.873348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:40.873348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:40.873348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:40:40.873348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:40.873348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:40.873348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.873348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:40.873682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:40.873682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:40.873682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.873682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:40.873682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:40.873682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:40.873682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:40.873682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:40.873682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:40.873682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:40.873682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:40.873682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:40.873682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:40.873682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:40.874065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:40.874065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:40.874065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.874065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:40.874409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.874409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:40.874409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:40.874409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:40.874409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:40.874409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:40.874409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.874409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:40.874409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:40.874409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.874409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:40.874409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:40.874409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.874409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:40.874409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:40.874409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.874816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:40.874816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:40.874816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:40:40.874816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:40.874816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:40.874816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.874816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:40.874816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:40.874816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:40.874816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.874816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:40.874816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:40.874816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:40.874816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:40.874816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:40.874816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:40.874816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:40.874816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:40.875116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:40.875116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:40.875116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:40.875116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:40.875116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:40.875116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.875116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:40.878570 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:40.878959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:40.879006 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:40.879110 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:40.879540 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:40.879853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.880410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:40.880410 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:40.880410 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:40.880410 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:40.880410 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:40.880410 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:40.880410 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.880410 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:40.881007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.881007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:40.881007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:40.881007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:40.881007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.881007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:40.881007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:40.881007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:40.881007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.881007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:40.881007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:40.881007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:40.881007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:40.881007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:40.881007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:40.881274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:40.881274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:40.881274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:40.881274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:40.881274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:40.881274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:40.881274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:40.881274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.881274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:40.881613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.881613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:40.881613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:40.881613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:40.881613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.881613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:40.881613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:40.881613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:40.881613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:40.881613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.881613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:40.882035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:40.882035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:40.882035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.882035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:40.882035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:40.882035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:40.882035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:40.882035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:40.882035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:40.882035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:40.882035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:40.882035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:40.882035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:40.882035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:40.882499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:40.882499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:40.882499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.882499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:40.882499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.882499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:40.882499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:40.882499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:40.882499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:40.882499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:40.882499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.882499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:40.882499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:40.882499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.882825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:40.882825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:40.882825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.882825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:40.882825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:40.882825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.882825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:40.882825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:40.882825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:40.882825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:40.882825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.882825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:40.882825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:40.882825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:40.882825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.882825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:40.882825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:40.882825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:40.882825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:40.883132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:40.883132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:40.883132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:40.883132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:40.883132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:40.883132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:40.883132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:40.883132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:40.883132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:40.883132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.883132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:40.896149 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:40.896427 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:40.896528 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:40.896692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:40.897007 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:40.897301 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:40.897526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.898031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:40.898031 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:40.898031 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:40.898031 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:40.898031 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:40.898031 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:40.898031 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.898031 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:40.898658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.898658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:40.898658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:40.898658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:40.898658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.898658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:40.898658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:40.898658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.898658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:40.898658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:40.898658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:40.898658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.898945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:40.898945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:40.898945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:40.898945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:40.898945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:40.898945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:40.898945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:40.898945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:40.898945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:40.898945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:40.898945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:40.898945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:40.898945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:40.899172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.899172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:40.899332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.899332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:40.899332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:40.899332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:40.899332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:40.899332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:40.899332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.899332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:40.899332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:40.899332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.899332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:40.899332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:40.899332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.899332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:40.899332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:40.899332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:40.899654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:40.899654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:40.899654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.899654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:40.899654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:40.899654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:40.899654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:40.899654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:40.899654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:40.899654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:40.899654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:40.899654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:40.899654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:40.899654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:40.899654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:40.899654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:40.899654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:40.899654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:40.899654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:40.900047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:40.900047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:40.900047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:43.782244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:43.782738 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:43.782738 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:43.783156 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:43.814393 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:43.814630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:43.815147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:43.815147 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:43.815147 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:43.815147 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:43.815147 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:43.815147 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:43.815147 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:43.815147 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:43.816016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:43.816016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:43.816016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:43.816016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:43.816016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:43.816016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:43.816016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:43.816016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:43.816016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:43.816016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:43.816016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:43.816016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:43.816856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:43.816856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:43.816856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:43.816856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:43.816856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:43.816856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:43.816856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:43.816856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:43.816856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:43.816856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:43.816856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:43.816856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:43.816856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:43.817964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:43.817964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:43.817964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:43.817964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:43.817964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:43.817964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:43.817964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:43.817964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:43.817964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:43.817964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:43.817964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:43.817964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:43.817964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:43.817964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:43.817964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:43.818363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:43.818363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:43.818363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:43.818363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:43.818363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:43.818363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:43.818363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:43.818363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:43.818363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:43.818363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:43.818363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:43.818363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:43.818363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:43.818363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:43.818363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:43.818363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:43.818363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:43.818363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:43.818363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:43.818858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:43.818858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:43.818858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:43.818858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:43.818858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:43.818858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:45.861455 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:45.862059 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:45.862059 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:45.862286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:45.862466 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:45.862626 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:45.862757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.863268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:45.863268 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:45.863268 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:45.863268 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:45.863268 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:45.863268 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:45.863268 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.863268 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:45.863942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.863942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:45.863942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:45.863942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:45.863942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.863942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:45.863942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:45.863942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.863942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:45.863942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:45.863942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:45.863942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:45.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:45.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:45.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:45.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:45.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:45.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:45.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:45.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:45.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:45.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:45.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:45.864242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:45.864586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.864586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:45.864922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.864922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:45.864922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:45.864922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:45.864922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:45.864922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:45.864922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.864922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:45.864922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:45.864922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.864922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:45.864922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:45.864922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.864922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:45.864922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:45.864922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.865288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:45.865288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:45.865288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.865288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:45.865288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:45.865288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:45.865288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.865288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:45.865288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:45.865288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:45.865288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:45.865288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:45.865288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:45.865288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:45.865288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:45.865288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:45.865288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:45.865288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:45.865288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:45.865802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:45.865802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.865802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:45.871331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:45.871331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:45.871331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:45.871331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:40:45.871331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:45.871991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.874941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:45.874941 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:45.874941 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:45.874941 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:45.874941 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:45.874941 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:45.874941 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.874941 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:45.874941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.874941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:45.874941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:45.874941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:45.874941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.874941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:45.874941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:45.874941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:45.874941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.874941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:45.875799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:45.875799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:45.875799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:45.875799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:45.875799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:45.875799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:45.875799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:45.875799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:45.875799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:45.875799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:45.875799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:45.875799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:45.875799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.875799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:45.875799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.876237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:45.876237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:45.876237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:45.876237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.876237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:45.876237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:45.876237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:40:45.876237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:45.876237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:45.876237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.876237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:45.876237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:45.876237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:45.876637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.876637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:45.876637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:45.876637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:45.876637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:45.876637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:45.876637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:45.876637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:45.876637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:45.876637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:45.876637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:45.876637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:45.876637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:45.876637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:45.877008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.877008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:45.877008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.877008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:45.877008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:45.877008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:45.877008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:45.877008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:45.877008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.877008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:45.877008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:45.877008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.877008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:45.877008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:45.877008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.877528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:45.877528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:45.877528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.877528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:45.877528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:45.877528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:40:45.877528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:45.877528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:45.877528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.877528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:45.877528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:45.877528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:45.877528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.877528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:45.877528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:45.877528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:45.877528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:45.877528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:45.877528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:45.878053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:45.878053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:45.878053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:45.878053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:45.878053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:45.878053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:45.878053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:45.878053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.878053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:45.878998 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:45.879092 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:45.879183 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:45.879309 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:45.879612 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:45.879851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.880326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:45.880326 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:45.880326 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:45.880326 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:45.880326 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:45.880326 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:45.880326 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.880326 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:45.880863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.880863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:45.880863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:45.880863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:45.880863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.880863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:45.880863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:45.880863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:45.880863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.880863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:45.880863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:45.880863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:45.880863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:45.880863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:45.880863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:45.881275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:45.881275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:45.881275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:45.881275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:45.881275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:45.881275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:45.881275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:45.881275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.881275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:45.881663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.881663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:45.881663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:45.881663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:45.881663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.881663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:45.881663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:45.881663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:45.881663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:45.881663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.881663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:45.881980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:45.881980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:45.881980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.881980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:45.881980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:45.881980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:45.881980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:45.881980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:45.881980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:45.881980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:45.881980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:45.881980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:45.881980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:45.881980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:45.883628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:45.883628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:45.883628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.883628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:45.883628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.883628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:45.883628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:45.883628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:45.883628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:45.883628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:45.883628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.883628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:45.883628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:45.883628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.884038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:45.884038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:45.884038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.884038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:45.884038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:45.884038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.884038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:45.884038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:45.884038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:45.884038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:45.884038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.884038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:45.884038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:45.884038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:45.884038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.884038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:45.884038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:45.884038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:45.884038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:45.884567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:45.884567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:45.884567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:45.884567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:45.884567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:45.884567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:45.884567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:45.884567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:45.884567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:45.884567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.884567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:45.896583 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:45.896709 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:45.896800 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:45.896934 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:45.897228 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:45.897485 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:45.897667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.898136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:45.898136 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:45.898136 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:45.898136 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:45.898136 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:45.898136 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:45.898136 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.898136 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:45.898883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.898883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:45.898883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:45.898883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:45.898883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.898883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:45.898883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:45.898883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.898883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:45.898883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:45.898883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:45.898883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.899208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:45.899208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:45.899208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:45.899208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:45.899208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:45.899208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:45.899208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:45.899208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:45.899208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:45.899208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:45.899208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:45.899208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:45.899208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:45.899554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.899554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:45.899874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.899874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:45.899874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:45.899874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:45.899874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:45.899874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:45.899874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.899874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:45.899874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:45.899874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.899874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:45.899874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:45.899874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.899874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:45.899874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:45.899874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:45.900301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:45.900301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:45.900301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.900301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:45.900301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:45.900301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:45.900301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:45.900301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:45.900301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:45.900301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:45.900301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:45.900301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:45.900301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:45.900301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:45.900301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:45.900301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:45.900301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:45.900301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:45.900301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:45.900860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:45.900860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:45.900860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:48.782558 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:48.782996 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:48.782996 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:48.783515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:48.814633 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:48.814855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:48.815410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:48.815410 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:48.815410 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:48.815410 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:48.815410 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:48.815410 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:48.815410 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:48.815410 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:48.816270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:48.816270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:48.816270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:48.816270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:48.816270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:48.816270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:48.816270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:48.816270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:48.816270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:48.816270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:48.816270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:48.816270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:48.816524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:48.816524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:48.816524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:48.816524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:48.816524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:48.816524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:48.816524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:48.816524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:48.816524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:48.816524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:48.816524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:48.816524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:48.816524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:48.816765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:48.816765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:48.817088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:48.817088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:48.817088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:48.817088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:48.817088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:48.817088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:48.817088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:48.817088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:48.817088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:48.817088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:48.817088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:48.817088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:48.817088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:48.817088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:48.817088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:48.817088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:48.817764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:48.817764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:48.817764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:48.817764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:48.817764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:48.817764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:48.817764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:48.817764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:48.817764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:48.817764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:48.817764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:48.817764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:48.817764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:48.817764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:48.817764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:48.817764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:48.817764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:48.817764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:48.817764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:48.818328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:48.818328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:48.818328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:50.861854 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:50.863190 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:50.863190 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:50.863563 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:50.863758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:50.863872 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:50.864001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.865890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:50.865890 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:50.865890 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:50.865890 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:50.865890 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:50.865890 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:50.865890 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.865890 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:50.865890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.865890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:50.865890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:50.865890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:50.865890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.865890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:50.865890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:50.866245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.866245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:50.866245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:50.866245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:50.866245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.866245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:50.866245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:50.866245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:50.866245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:50.866245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:50.866245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:50.866245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:50.866245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:50.866245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:50.866492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:50.866492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:50.866492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:50.866492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:50.866492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.866492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:50.866492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.866492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:50.866492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:50.866492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:50.866492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:50.866492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:50.866492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.866492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:50.866764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:50.867076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:50.867076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:50.867076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:50.867076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:50.867076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:50.867076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:50.867076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:50.867076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:50.867076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.867076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:50.870793 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:50.871121 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:50.871317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:50.871505 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:40:50.871987 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:50.872181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.875170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:50.875170 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:50.875170 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:50.875170 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:50.875170 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:50.875170 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:50.875170 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.875170 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:50.875170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.875170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:50.875170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:50.875170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:50.875170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.875170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:50.875170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:50.875170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:50.875170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.875170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:50.875704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:50.875704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:50.875704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:50.875704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:50.875704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:50.875704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:50.875704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:50.875704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:50.875704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:50.875704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:50.875704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:50.875704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:50.875704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.875704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:50.875704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.876082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:50.876082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:50.876082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:50.876082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.876082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:50.876082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:50.876082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:40:50.876082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:50.876082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:50.876082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.876082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:50.876082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:50.876082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:50.876506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.876506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:50.876506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:50.876506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:50.876506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:50.876506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:50.876506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:50.876506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:50.876506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:50.876506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:50.876506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:50.876506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:50.876506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:50.876506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:50.876910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.876910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:50.876910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.876910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:50.876910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:50.876910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:50.876910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:50.876910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:50.876910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.876910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:50.876910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:50.876910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.876910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:50.876910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:50.876910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.877335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:50.877335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:50.877335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.877335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:50.877335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:50.877335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:40:50.877335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:50.877335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:50.877335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.877335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:50.877335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:50.877335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:50.877335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.877335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:50.877335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:50.877335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:50.877335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:50.877335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:50.877335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:50.877847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:50.877847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:50.877847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:50.877847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:50.877847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:50.877847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:50.877847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:50.877847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.877847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:50.879186 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:50.879488 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:50.879671 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:50.879860 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:50.880262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:50.880460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:50.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:50.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:50.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:50.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:50.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:50.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:50.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:50.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:50.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:50.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:50.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:50.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:50.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:50.884296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:50.884296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:50.884296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:50.884296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:50.884296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:50.884296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:50.884296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:50.884296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:50.884296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:50.884296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:50.884296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:50.884296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:50.884296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.884296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:50.884296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.884749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:50.884749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:50.884749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:50.884749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.884749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:50.884749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:50.884749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:50.884749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:50.884749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.884749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:50.884749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:50.884749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:50.884749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.885071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:50.885071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:50.885071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:50.885071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:50.885071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:50.885071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:50.885071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:50.885071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:50.885071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:50.885071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:50.885071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:50.885071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:50.885071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:50.885433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.885433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:50.885433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.885433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:50.885433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:50.885433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:50.885433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:50.885433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:50.885433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.885433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:50.885433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:50.885433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.885433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:50.885433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:50.885433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.885829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:50.885829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:50.885829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.885829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:50.885829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:50.885829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:50.885829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:50.885829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.885829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:50.885829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:50.885829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:50.885829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.885829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:50.885829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:50.885829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:50.885829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:50.885829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:50.885829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:50.885829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:50.886361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:50.886361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:50.886361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:50.886361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:50.886361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:50.886361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:50.886361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.886361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:50.896949 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:50.897284 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:50.897503 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:50.897736 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:50.898074 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:50.898265 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:50.898397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.900135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:50.900135 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:50.900135 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:50.900135 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:50.900135 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:50.900135 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:50.900135 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.900135 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:50.900135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.900135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:50.900135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:50.900135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:50.900135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.900135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:50.900135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:50.900444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.900444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:50.900444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:50.900444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:50.900444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.900444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:50.900444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:50.900444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:50.900444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:50.900444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:50.900444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:50.900444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:50.900444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:50.900444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:50.900753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:50.900753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:50.900753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:50.900753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:50.900753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.900753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:50.900753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.900753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:50.900753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:50.900753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:50.900753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:50.900753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:50.900753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.900753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:50.901092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:50.901413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:50.901413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:50.901413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:50.901413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:50.901413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:50.901413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:50.901413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:50.901413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:50.901413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:50.901413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:53.782962 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:53.783440 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:53.783440 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:53.784342 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:53.793142 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=241 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:40:53.815806 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:53.815806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:53.815957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:53.815957 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:53.815957 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:53.815957 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:53.815957 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:53.815957 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:53.815957 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:53.815957 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:53.816578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:53.816578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:53.816578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:53.816578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:53.816578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:53.816578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:53.816578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:53.816578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:53.816578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:53.816578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:53.816578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:53.816578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:53.816854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:53.816854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:53.816854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:53.816854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:53.816854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:53.816854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:53.816854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:53.816854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:53.816854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:53.816854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:53.816854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:53.816854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:53.816854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:53.817045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:53.817045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:53.817307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:53.817307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:53.817307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:53.817307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:53.817307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:53.817307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:53.817307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:53.817307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:53.817307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:53.817307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:53.817307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:53.817307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:53.817307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:53.817307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:53.817307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:53.817307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:53.817770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:53.817770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:53.817770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:53.817770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:53.817770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:53.817770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:53.817770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:53.817770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:53.817770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:53.817770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:53.817770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:53.817770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:53.817770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:53.817770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:53.817770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:53.817770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:53.817770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:53.817770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:53.817770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:53.818209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:53.818209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:53.818209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:55.862857 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:55.863507 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:55.863507 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:55.864879 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:55.864879 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:55.864879 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:55.864879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.864994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:55.864994 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:55.864994 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:55.864994 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:55.864994 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:55.864994 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:55.864994 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.864994 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:55.865776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.865776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:55.865776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:55.865776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:55.865776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.865776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:55.865776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:55.865776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.865776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:55.865776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:55.865776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:55.865776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:55.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:55.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:55.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:55.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:55.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:55.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:55.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:55.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:55.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:55.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:55.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:55.866382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:55.866642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.866642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:55.866642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.866642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:55.866642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:55.866642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:55.866642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:55.866642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:55.866642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.866642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:55.866642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:55.866642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.866642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:55.866642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:55.866642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.866954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:55.866954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:55.866954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.866954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:40:55.866954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:55.866954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.866954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:55.866954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:55.866954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:55.866954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.866954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:55.866954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:55.866954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:55.866954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:55.866954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:55.866954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:55.866954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:55.866954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:55.866954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:55.867307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:55.867307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:55.867307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:55.867307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:55.867307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.867307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:55.871222 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:55.871508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:55.871646 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:55.871795 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:40:55.872735 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:55.872735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.872894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:55.872894 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:55.872894 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:55.872894 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:55.872894 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:55.872894 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:55.872894 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.872894 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:55.873442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.873442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:55.873442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:55.873442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:55.873442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.873442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:55.873442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:55.873442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:55.873442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.873442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:55.873442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:55.873442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:55.873442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:55.873442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:55.873442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:55.874360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:55.874360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:55.874360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:55.874360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:55.874360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:55.874360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:55.874360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:55.874360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.874360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:55.874360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.874360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:55.874360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:55.874360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:55.874360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.875407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:55.875407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:55.875407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:40:55.875407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:55.875407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:55.875407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.875407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:55.875407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:55.875407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:55.875407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.875407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:55.875407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:55.875407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:55.875752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:55.875752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:55.875752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:55.875752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:55.875752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:55.875752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:55.875752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:55.875752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:55.875752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:55.875752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:55.875752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.875752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:55.875933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.875933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:55.875933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:55.875933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:55.875933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:55.875933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:55.875933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.875933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:55.875933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:55.875933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.875933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:55.875933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:55.875933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.875933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:55.875933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:55.875933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.876233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:40:55.876233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:40:55.876233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:40:55.876233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:55.876233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:55.876233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.876233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:55.876233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:55.876233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:55.876233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.876233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:55.876233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:55.876233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:55.876233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:55.876233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:55.876233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:55.876233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:55.876233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:55.876540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:55.876540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:55.876540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:55.876540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:55.876540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:55.876540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.876540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:55.879552 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:55.879749 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:55.879822 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:55.879924 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:55.880666 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:55.880666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.880826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:55.880826 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:55.880826 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:55.880826 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:55.880826 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:55.880826 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:55.880826 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.880826 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:55.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:40:55.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:55.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:55.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:55.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:55.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:40:55.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:40:55.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:55.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:40:55.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:55.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:40:55.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:40:55.881809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:55.881809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:40:55.881809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:55.881809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:40:55.881809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:40:55.881809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:55.881809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:40:55.881809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.881809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:40:55.881942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.881942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:55.881942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:55.881942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:55.881942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.881942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:55.881942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:55.881942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:55.881942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:55.881942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.881942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:55.882143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:55.882143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:55.882143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.882143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:55.882143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:55.882143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:55.882143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:55.882143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:55.882143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:55.882143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:55.882143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:55.882143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:55.882143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:55.882143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:55.882378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:55.882378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:55.882378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.882378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:55.882793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.882793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:55.882793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:55.882793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:55.882793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:55.882793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:55.882793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.882793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:55.882793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:55.882793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.882793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:55.882793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:55.882793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.882793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:55.882793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:55.882793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.882793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:40:55.883673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:40:55.883673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:40:55.883673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:40:55.883673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.883673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:55.883673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:55.883673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:55.883673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.883673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:55.883673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:55.883673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:55.883673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:55.883673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:55.883673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:55.883673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:55.883673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:55.883673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:55.883673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:55.883673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:55.884124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:55.884124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:55.884124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.884124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:55.897455 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:55.897750 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:55.897826 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:55.897953 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:55.898933 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:55.898933 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:55.898933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.899117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:55.899117 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:55.899117 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:55.899117 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:55.899117 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:55.899117 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:55.899117 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.899117 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:55.899578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.899578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:55.899578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:55.899578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:55.899578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.899578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:55.899578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:55.899578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.899578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:55.899578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:55.899578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:55.899578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.899823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:55.899823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:55.899823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:55.899823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:55.899823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:55.899823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:55.899823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:55.899823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:55.899823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:55.899823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:55.899823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:55.899823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:55.899823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:55.900025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.900025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:55.900313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.900313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:55.900313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:55.900313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:55.900313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:55.900313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:55.900313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.900313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:55.900313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:55.900313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.900313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:55.900313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:55.900313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.900313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:55.900313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:55.900313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:55.901110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:40:55.901110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:55.901110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.901110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:55.901110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:55.901110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:55.901110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:55.901110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:55.901110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:55.901110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:55.901110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:55.901110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:55.901110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:55.901110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:55.901110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:55.901110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:55.901110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:55.901110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:55.901110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:55.901519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:55.901519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:55.901519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:58.783447 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:58.783901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:58.783901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:58.784344 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:40:58.810162 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:58.810471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.810965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:58.810965 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:58.810965 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:58.810965 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:58.810965 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:58.810965 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:58.810965 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.810965 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:58.811830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.811830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:58.811830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:58.811830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:40:58.811830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:58.811830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:58.811830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:58.811830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:58.811830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:58.811830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:58.811830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:58.811830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:58.812049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:58.812049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:58.812049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:58.812049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:58.812049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:58.812049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:58.812049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:58.812049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:58.812049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:58.812049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:58.812049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:58.812049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:58.812049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:58.812280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.812280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:58.812715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.812715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:40:58.812715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:40:58.812715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:40:58.812715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:40:58.812715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:40:58.812715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:58.812715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:40:58.812715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:40:58.812715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:58.812715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:40:58.812715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:40:58.812715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:40:58.812715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:40:58.812715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:40:58.812715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:40:58.813179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:40:58.813179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:58.813179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:58.813179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:58.813179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:58.813179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:40:58.813179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:58.813179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:40:58.813179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:58.813179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:40:58.813179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:58.813179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:40:58.813179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:40:58.813179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:58.813179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:40:58.813179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:58.813179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:40:58.813179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:40:58.813179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:58.813500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:40:58.813500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.813500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:40:58.860056 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:40:58.860333 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:40:58.860468 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:40:58.867442 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 08 14:40:58.867582 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Number of inconsistencies found at create/update: networks=5, security_groups=1, routers=2, router_ports=2 {{(pid=89274) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 14:40:58.867651 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=9 {{(pid=89274) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 14:40:58.867826 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource a137566a-eaf1-4f5d-a192-15d63e6c049a (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:40:58.896708 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-a137566a-eaf1-4f5d-a192-15d63e6c049a, resource={'id': 'a137566a-eaf1-4f5d-a192-15d63e6c049a', 'name': 'private', 'tenant_id': 'ba473288446843d3a29e25267968ba61', 'project_id': 'ba473288446843d3a29e25267968ba61', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['7b3cb50f-b942-4054-b735-5f7be989b8f8', 'ba10f097-6c2b-4188-8d42-e9415eff0039'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:20:42Z', 'updated_at': '2026-06-08T14:20:44Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 26551}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:58.896977 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-a137566a-eaf1-4f5d-a192-15d63e6c049a, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:58.897205 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-c4b70b51-9d46-4e6b-a1e7-b3e6a6b8b666, external_ids_update=None, columns={'tag': 26551}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:58.897346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.897857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:58.897857 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:58.897857 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:58.897857 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:58.897857 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:58.897857 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:58.897857 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.897857 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:58.898751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource a137566a-eaf1-4f5d-a192-15d63e6c049a (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.898751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:58.898751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:40:58.898751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:40:58.898751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:40:58.898751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:40:58.898751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:40:58.898751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:58.898751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:58.898751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:58.898751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:58.898980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:58.898980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:58.898980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:58.898980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:58.898980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:58.898980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:58.898980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:58.898980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:58.898980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:58.898980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:58.898980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:58.898980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:58.898980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:58.898980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:58.899194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:58.899194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.899194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:58.899194 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource fffd4e9d-b343-4a97-8e5f-4015f805fc46 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:40:58.923052 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, resource={'id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'name': 'public', 'tenant_id': 'ba473288446843d3a29e25267968ba61', 'project_id': 'ba473288446843d3a29e25267968ba61', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['4ee83802-2181-432f-a76d-d3335bbf4e0b', 'b9859e49-8026-492d-a0a4-11326477713e'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-08T14:20:46Z', 'updated_at': '2026-06-08T14:20:52Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:58.923280 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:58.923621 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-0817ce61-1c85-4457-911a-0b8fdccf389d, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:58.923800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.924313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:58.924313 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:58.924313 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:58.924313 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:58.924313 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:58.924313 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:58.924313 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.924313 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:58.925064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource fffd4e9d-b343-4a97-8e5f-4015f805fc46 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.925064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:58.925064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:40:58.925064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:40:58.925064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:40:58.925064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:40:58.925064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:40:58.925064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:58.925064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:58.925064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:58.925064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:58.925349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:58.925349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:58.925349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:58.925349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:58.925349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:58.925349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:58.925349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:58.925349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:58.925349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:58.925349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:58.925349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:58.925349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:58.925349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:58.925349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:58.925622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:58.925622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.925622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:58.925622 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 73426422-a7fc-44cd-9fc5-cd2d8fa95c38 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:40:58.948337 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-73426422-a7fc-44cd-9fc5-cd2d8fa95c38, resource={'id': '73426422-a7fc-44cd-9fc5-cd2d8fa95c38', 'name': 'shared', 'tenant_id': 'ba473288446843d3a29e25267968ba61', 'project_id': 'ba473288446843d3a29e25267968ba61', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['2b53474a-74e5-4c44-bb1b-86c8215bd7ab'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:21:52Z', 'updated_at': '2026-06-08T14:21:52Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 292}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:58.948513 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-73426422-a7fc-44cd-9fc5-cd2d8fa95c38, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:58.948729 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-5c6a587e-b2f4-42c7-9706-8a2e20b0df46, external_ids_update=None, columns={'tag': 292}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:58.948848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.949264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:58.949264 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:58.949264 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:58.949264 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:58.949264 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:58.949264 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:58.949264 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.949264 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:58.950010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 73426422-a7fc-44cd-9fc5-cd2d8fa95c38 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.950010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:58.950010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:40:58.950010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:40:58.950010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:40:58.950010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:40:58.950010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:40:58.950010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:58.950010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:58.950010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:58.950010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:58.950286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:58.950286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:58.950286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:58.950286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:58.950286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:58.950286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:58.950286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:58.950286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:58.950286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:58.950286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:58.950286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:58.950286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:58.950286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:58.950286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:58.950514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:58.950514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.950514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:58.950514 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 0fd61d65-2424-4180-8dde-86d9f980cb55 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:40:58.973069 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-0fd61d65-2424-4180-8dde-86d9f980cb55, resource={'id': '0fd61d65-2424-4180-8dde-86d9f980cb55', 'name': 'tempest-TestStampPattern-1462761963-network', 'tenant_id': 'd0527bfd7798485a8675cab8bb0fefd9', 'project_id': 'd0527bfd7798485a8675cab8bb0fefd9', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['90afd701-def4-482b-ba04-3f02d965d593'], 'standard_attr_id': 1531, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:39:11Z', 'updated_at': '2026-06-08T14:39:11Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 34145}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:58.973275 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-0fd61d65-2424-4180-8dde-86d9f980cb55, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestStampPattern-1462761963-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:58.973481 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-5a366f1f-55dd-4421-90df-7e9965bba3ff, external_ids_update=None, columns={'tag': 34145}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:58.973559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.973959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:58.973959 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:58.973959 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:58.973959 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:58.973959 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:58.973959 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:58.973959 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.973959 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:58.974972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 0fd61d65-2424-4180-8dde-86d9f980cb55 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.974972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:58.974972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:40:58.974972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:40:58.974972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:40:58.974972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:40:58.974972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:40:58.974972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:58.974972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:58.974972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:58.974972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:58.975355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:58.975355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:58.975355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:58.975355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:58.975355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:58.975355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:58.975355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:58.975355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:58.975355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:58.975355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:58.975355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:58.975355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:58.975355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:58.975355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:58.975716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:58.975716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.975716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:58.975716 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource be01914e-f844-4b9b-b839-53c33541205b (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:40:58.997470 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-be01914e-f844-4b9b-b839-53c33541205b, resource={'id': 'be01914e-f844-4b9b-b839-53c33541205b', 'name': 'tempest-TestStampPattern-21296117-network', 'tenant_id': '282bfe2674c940b5aaeb0174d9d2242e', 'project_id': '282bfe2674c940b5aaeb0174d9d2242e', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['ca2ab66e-fdcf-416d-b751-3c9c9d12fb78'], 'standard_attr_id': 1544, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:39:15Z', 'updated_at': '2026-06-08T14:39:15Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 47080}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:58.997680 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-be01914e-f844-4b9b-b839-53c33541205b, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestStampPattern-21296117-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:58.997865 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-310ade98-98db-45a2-aa6f-d6ed0106b43c, external_ids_update=None, columns={'tag': 47080}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:58.997953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.998401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:58.998401 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:58.998401 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:58.998401 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:58.998401 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:58.998401 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:58.998401 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.998401 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:58.999397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource be01914e-f844-4b9b-b839-53c33541205b (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:58.999397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:58.999397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:40:58.999397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:40:58.999397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:40:58.999397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:40:58.999397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:40:58.999397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:58.999397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:58.999397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:58.999397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:58.999761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:58.999761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:58.999761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:58.999761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:58.999761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:58.999761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:58.999761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:58.999761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:58.999761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:58.999761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:58.999761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:58.999761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:58.999761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:58.999761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:59.000110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:59.000110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:59.000110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:59.000110 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource ebe2e1fa-d659-4180-b0ab-c5ab9a8368f1 (type: security_groups) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:40:59.027071 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Successfully bumped revision number for resource ebe2e1fa-d659-4180-b0ab-c5ab9a8368f1 (type: security_groups) to 7 Jun 08 14:40:59.027190 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 3df748d4-c112-4f5e-9b9f-8fba3dd8da91 (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:40:59.084531 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, resource={'id': '3df748d4-c112-4f5e-9b9f-8fba3dd8da91', 'name': 'tempest-TestStampPattern-1462761963-router', 'project_id': 'd0527bfd7798485a8675cab8bb0fefd9', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.24'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::10a'}], 'enable_snat': True}, 'gw_port_id': '34de5e3d-8039-4618-91f1-9d1c0f30817b', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.24', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::10a', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:39:11Z', 'updated_at': '2026-06-08T14:39:13Z', 'revision_number': 3, 'tenant_id': 'd0527bfd7798485a8675cab8bb0fefd9'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:59.084865 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, columns={'external_ids': {'neutron:router_name': 'tempest-TestStampPattern-1462761963-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:59.085007 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "34de5e3d-8039-4618-91f1-9d1c0f30817b"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:59.085124 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "34de5e3d-8039-4618-91f1-9d1c0f30817b"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:59.085248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:59.085693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:59.085693 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:59.085693 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:59.085693 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:59.085693 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:59.085693 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:59.085693 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:59.085693 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:59.085862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router 3df748d4-c112-4f5e-9b9f-8fba3dd8da91. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:59.086771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 3df748d4-c112-4f5e-9b9f-8fba3dd8da91 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:59.086771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:59.086771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:40:59.086771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:40:59.086771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:40:59.086771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:40:59.086771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:40:59.086771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:40:59.086771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:40:59.086771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:40:59.086771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:40:59.086984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:40:59.086984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:40:59.086984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:59.086984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:59.086984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:59.086984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:59.086984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:59.086984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:59.086984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:59.086984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:59.086984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:59.086984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:59.086984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:59.086984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:59.087221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:59.087221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:59.087221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:59.087221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:59.087221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:59.087221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:59.087221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:59.087221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:59.087221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:59.087221 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 459211a9-6ff5-45d1-9ad2-0aff99e451a4 (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:40:59.146714 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, resource={'id': '459211a9-6ff5-45d1-9ad2-0aff99e451a4', 'name': 'tempest-TestStampPattern-21296117-router', 'project_id': '282bfe2674c940b5aaeb0174d9d2242e', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.166'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::8e'}], 'enable_snat': True}, 'gw_port_id': '2d70280f-c1ff-40ec-95b8-6a1bcc85f1c3', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.166', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::8e', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:39:16Z', 'updated_at': '2026-06-08T14:39:18Z', 'revision_number': 3, 'tenant_id': '282bfe2674c940b5aaeb0174d9d2242e'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:59.146940 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, columns={'external_ids': {'neutron:router_name': 'tempest-TestStampPattern-21296117-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:59.147180 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "2d70280f-c1ff-40ec-95b8-6a1bcc85f1c3"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:59.147340 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "2d70280f-c1ff-40ec-95b8-6a1bcc85f1c3"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:59.147340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:59.147778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:59.147778 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:59.147778 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:59.147778 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:59.147778 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:59.147778 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:59.147778 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:59.147778 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:59.148026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router 459211a9-6ff5-45d1-9ad2-0aff99e451a4. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:59.148933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 459211a9-6ff5-45d1-9ad2-0aff99e451a4 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:59.148933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:59.148933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:40:59.148933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:40:59.148933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:40:59.148933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:40:59.148933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:40:59.148933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:40:59.148933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:40:59.148933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:40:59.148933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:40:59.149273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:40:59.149273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:40:59.149273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:59.149273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:59.149273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:59.149273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:59.149273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:59.149273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:59.149273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:59.149273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:59.149273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:59.149273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:59.149273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:59.149273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:59.149636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:59.149636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:59.149636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:59.149636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:59.149636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:59.149636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:59.149636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:59.149636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:59.149636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:59.149636 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 34de5e3d-8039-4618-91f1-9d1c0f30817b (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:40:59.224656 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-34de5e3d-8039-4618-91f1-9d1c0f30817b, resource={'id': '34de5e3d-8039-4618-91f1-9d1c0f30817b', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:c1:5f:e2', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '3df748d4-c112-4f5e-9b9f-8fba3dd8da91', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1536, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.24'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::10a'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:39:12Z', 'updated_at': '2026-06-08T14:39:14Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:59.224904 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-34de5e3d-8039-4618-91f1-9d1c0f30817b, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.24/24', '2001:db8::10a/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:59.225170 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=34de5e3d-8039-4618-91f1-9d1c0f30817b, lrouter_port=lrp-34de5e3d-8039-4618-91f1-9d1c0f30817b, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:59.225421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:59.225901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:59.225901 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:59.225901 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:59.225901 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:59.225901 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:59.225901 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:59.225901 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:59.225901 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:59.226583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 34de5e3d-8039-4618-91f1-9d1c0f30817b (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:59.226583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:59.226583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:40:59.226583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:40:59.226583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:40:59.226583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:40:59.226583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:40:59.226583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:59.226583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:59.226583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:59.226583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:59.226827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:59.226827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:59.226827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:59.226827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:59.226827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:59.226827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:59.226827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:59.226827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:59.226827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:59.226827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:59.226827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:59.226827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:59.226827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:59.226827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:59.227029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:59.227029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:59.227029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:59.227029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 2d70280f-c1ff-40ec-95b8-6a1bcc85f1c3 (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:40:59.301879 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-2d70280f-c1ff-40ec-95b8-6a1bcc85f1c3, resource={'id': '2d70280f-c1ff-40ec-95b8-6a1bcc85f1c3', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:76:95:06', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '459211a9-6ff5-45d1-9ad2-0aff99e451a4', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1549, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.166'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::8e'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:39:16Z', 'updated_at': '2026-06-08T14:39:20Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:59.302107 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-2d70280f-c1ff-40ec-95b8-6a1bcc85f1c3, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.166/24', '2001:db8::8e/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:59.302421 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=2d70280f-c1ff-40ec-95b8-6a1bcc85f1c3, lrouter_port=lrp-2d70280f-c1ff-40ec-95b8-6a1bcc85f1c3, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:40:59.302653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:59.303157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:40:59.303157 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:59.303157 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:40:59.303157 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:40:59.303157 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:59.303157 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:40:59.303157 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:59.303157 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:40:59.303874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 2d70280f-c1ff-40ec-95b8-6a1bcc85f1c3 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:59.303874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:40:59.303874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:40:59.303874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:40:59.303874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:40:59.303874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:40:59.303874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:40:59.303874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:40:59.303874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:59.303874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:59.303874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:40:59.304080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:40:59.304080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:40:59.304080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:40:59.304080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:40:59.304080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:40:59.304080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:40:59.304080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:40:59.304080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:40:59.304080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:40:59.304080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:40:59.304080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:40:59.304080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:40:59.304080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:40:59.304080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:40:59.304325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:40:59.304325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:40:59.304325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:40:59.304325 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 0c296687-b996-4844-bd7e-67b87e0698ed (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:40:59.304325 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(0c296687-b996-4844-bd7e-67b87e0698ed) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:40:59.308830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 14a15d54-608f-4ef7-b427-26ff72b78b61 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:40:59.309028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(14a15d54-608f-4ef7-b427-26ff72b78b61) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:40:59.312679 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 14ded302-d74c-4447-8ff0-4d05ee4c5240 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:40:59.312845 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(14ded302-d74c-4447-8ff0-4d05ee4c5240) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:40:59.316869 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 3bb3cdcc-cd4a-4184-a4fa-9970ef30c898 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:40:59.317062 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(3bb3cdcc-cd4a-4184-a4fa-9970ef30c898) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:40:59.320950 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 47a298a9-f54c-40f5-b057-6668c1e5ec47 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:40:59.321151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(47a298a9-f54c-40f5-b057-6668c1e5ec47) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:40:59.324879 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 7267af20-8b2d-41e1-9231-b1edae50a716 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:40:59.325082 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(7267af20-8b2d-41e1-9231-b1edae50a716) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:40:59.328973 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 920851db-ff58-42c4-8d2b-3fcfab52c68a (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:40:59.329158 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(920851db-ff58-42c4-8d2b-3fcfab52c68a) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:40:59.332699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource c2cbf5dc-6f5b-4635-b67b-c9db491447b1 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:40:59.332908 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(c2cbf5dc-6f5b-4635-b67b-c9db491447b1) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:40:59.336467 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource e05217ee-370e-449b-bf1e-271670aa2c1c (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:40:59.336590 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(e05217ee-370e-449b-bf1e-271670aa2c1c) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:40:59.340377 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_for_inconsistencies finished in 0.480 seconds Jun 08 14:41:00.863303 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:00.863997 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:00.863997 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:00.864349 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:00.864510 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:00.864673 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:00.864789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.865275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:00.865275 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:00.865275 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:00.865275 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:00.865275 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:00.865275 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:00.865275 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.865275 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:00.865895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.865895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:00.865895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:00.865895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:00.865895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.865895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:00.865895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:00.865895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.865895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:00.865895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:00.865895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:00.865895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.866832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:00.866832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:00.866832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:00.866832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:00.866832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:00.866832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:00.866832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:00.866832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:00.866832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:00.866832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:00.866832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:00.866832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:00.866832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:00.867815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.867815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:00.867815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.867815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:00.867815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:00.867815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:00.867815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:00.867815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:00.867815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.867815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:00.867815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:00.867815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.867815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:00.867815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:00.867815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.868782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:00.868782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:00.868782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.868782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:00.868782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:00.868782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.868782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:00.868782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:00.868782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:00.868782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.868782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:00.868782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:00.868782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:00.868782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:00.868782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:00.868782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:00.868782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:00.868782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:00.868782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:00.869655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:00.869655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:00.869655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:00.869655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:00.869655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.869655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:00.871554 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:00.871802 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:00.871915 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:00.872036 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:41:00.872382 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:00.872569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.873080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:00.873080 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:00.873080 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:00.873080 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:00.873080 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:00.873080 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:00.873080 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.873080 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:00.873868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.873868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:00.873868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:00.873868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:00.873868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.873868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:00.873868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:00.873868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:00.873868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.873868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:00.873868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:00.873868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:00.873868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:00.873868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:00.873868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:00.874116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:00.874116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:00.874116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:00.874116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:00.874116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:00.874116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:00.874116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:00.874116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.874116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:00.874773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.874773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:00.874773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:00.874773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:00.874773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.874773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:00.874773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:00.874773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:41:00.874773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:00.874773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:00.874773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.874773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:00.874991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:00.874991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:00.874991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.874991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:00.874991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:00.874991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:00.874991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:00.874991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:00.874991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:00.874991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:00.874991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:00.874991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:00.874991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:00.874991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:00.875201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:00.875201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:00.875201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.875201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:00.875845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.875845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:00.875845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:00.875845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:00.875845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:00.875845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:00.875845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.875845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:00.875845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:00.875845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.875845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:00.875845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:00.875845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.875845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:00.875845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:00.875845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.876096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:00.876096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:00.876096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:41:00.876096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:00.876096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:00.876096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.876096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:00.876096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:00.876096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:00.876096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.876096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:00.876096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:00.876096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:00.876096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:00.876096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:00.876096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:00.876096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:00.876096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:00.876384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:00.876384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:00.876384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:00.876384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:00.876384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:00.876384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.876384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:00.879836 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:00.880027 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:00.880144 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:00.880299 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:00.880591 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:00.880811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.881245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:00.881245 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:00.881245 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:00.881245 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:00.881245 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:00.881245 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:00.881245 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.881245 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:00.882007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.882007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:00.882007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:00.882007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:00.882007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.882007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:00.882007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:00.882007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:00.882007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.882007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:00.882007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:00.882007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:00.882007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:00.882007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:00.882007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:00.882242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:00.882242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:00.882242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:00.882242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:00.882242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:00.882242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:00.882242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:00.882242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.882242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:00.882923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.882923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:00.882923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:00.882923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:00.882923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.882923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:00.882923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:00.882923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:00.882923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:00.882923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.882923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:00.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:00.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:00.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:00.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:00.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:00.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:00.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:00.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:00.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:00.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:00.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:00.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:00.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:00.883353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:00.883353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:00.883353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.883353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:00.884037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.884037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:00.884037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:00.884037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:00.884037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:00.884037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:00.884037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.884037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:00.884037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:00.884037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.884037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:00.884037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:00.884037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.884037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:00.884037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:00.884037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.884037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:00.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:00.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:00.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:00.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:00.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:00.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:00.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:00.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:00.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:00.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:00.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:00.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:00.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:00.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:00.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:00.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:00.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:00.884646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:00.884646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:00.884646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.884646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:00.897753 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:00.897944 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:00.897993 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:00.898121 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:00.898406 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:00.898638 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:00.898796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.899217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:00.899217 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:00.899217 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:00.899217 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:00.899217 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:00.899217 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:00.899217 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.899217 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:00.899810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.899810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:00.899810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:00.899810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:00.899810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.899810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:00.899810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:00.899810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.899810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:00.899810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:00.899810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:00.899810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.900022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:00.900022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:00.900022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:00.900022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:00.900022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:00.900022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:00.900022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:00.900022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:00.900022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:00.900022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:00.900022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:00.900022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:00.900022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:00.900218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.900218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:00.900491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.900491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:00.900491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:00.900491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:00.900491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:00.900491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:00.900491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.900491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:00.900491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:00.900491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.900491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:00.900491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:00.900491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.900491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:00.900491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:00.900491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:00.901243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:00.901243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:00.901243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.901243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:00.901243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:00.901243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:00.901243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:00.901243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:00.901243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:00.901243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:00.901243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:00.901243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:00.901243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:00.901243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:00.901243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:00.901243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:00.901243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:00.901243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:00.901243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:00.901617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:00.901617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:00.901617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:03.783899 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:03.785831 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:03.785831 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:03.785831 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:03.814149 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:03.814440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:03.815549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:03.815549 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:03.815549 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:03.815549 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:03.815549 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:03.815549 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:03.815549 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:03.815549 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:03.817490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:03.817490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:03.817490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:03.817490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:03.817490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:03.817490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:03.817490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:03.817490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:03.817490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:03.817490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:03.817490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:03.817490 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:03.817821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:03.817821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:03.817821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:03.817821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:03.817821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:03.817821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:03.817821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:03.817821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:03.817821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:03.817821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:03.817821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:03.817821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:03.817821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:03.818070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:03.818070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:03.818070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:03.818070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:03.818070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:03.818070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:03.818070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:03.818070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:03.818070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:03.818070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:03.818070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:03.818070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:03.818070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:03.818070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:03.818070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:03.818346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:03.818346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:03.818346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:03.818346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:03.818346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:03.818346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:03.818346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:03.818346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:03.818346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:03.818346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:03.818346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:03.818346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:03.818346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:03.818346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:03.818346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:03.818346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:03.818346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:03.818346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:03.818346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:03.818739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:03.818739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:03.818739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:03.818739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:03.818739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:03.818739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:05.864125 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:05.864125 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:05.864125 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:05.864883 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:05.865204 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:05.865363 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:05.865488 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1742b4f6-e341-446e-90b3-d2893aeee352, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:05.865610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.866115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:05.866115 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:05.866115 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:05.866115 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:05.866115 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:05.866115 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:05.866115 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.866115 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:05.866921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.866921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:05.866921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:05.866921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:05.866921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.866921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:05.866921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:05.866921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.866921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:05.866921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:05.866921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:05.866921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.867185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:05.867185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:05.867185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:05.867185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:05.867185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:05.867185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:05.867185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:05.867185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:05.867185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:05.867185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:05.867185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:05.867185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:05.867185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:05.867454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.867454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:05.867621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.867621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:05.867621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:05.867621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:05.867621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:05.867621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:05.867621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.867621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:05.867621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:05.867621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.867621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:05.867621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:05.867621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.867621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:05.867621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:05.867621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.868011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:05.868011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:05.868011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.868011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:05.868011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:05.868011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:05.868011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.868011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:05.868011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:05.868011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:05.868011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:05.868011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:05.868011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:05.868011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:05.868011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:05.868011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:05.868011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:05.868011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:05.868011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:05.868401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:05.868401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.868401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:05.871863 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:05.872069 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:05.872154 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:05.872245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:41:05.873104 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:05.873314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.873924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:05.873924 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:05.873924 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:05.873924 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:05.873924 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:05.873924 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:05.873924 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.873924 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:05.874551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.874551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:05.874551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:05.874551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:05.874551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.874551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:05.874551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:05.874551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:05.874551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.874551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:05.874551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:05.874551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:05.874551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:05.874551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:05.874551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:05.874865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:05.874865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:05.874865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:05.874865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:05.874865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:05.874865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:05.874865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:05.874865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.874865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:05.875205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.875205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:05.875205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:05.875205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:05.875205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.875205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:05.875205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:05.875205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:41:05.875205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:05.875205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:05.875205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.875205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:05.875455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:05.875455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:05.875455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.875455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:05.875455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:05.875455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:05.875455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:05.875455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:05.875455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:05.875455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:05.875455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:05.875455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:05.875455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:05.875455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:05.875699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:05.875699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:05.875699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.875699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:05.875960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.875960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:05.875960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:05.875960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:05.875960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:05.875960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:05.875960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.875960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:05.875960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:05.875960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.875960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:05.875960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:05.875960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.875960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:05.875960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:05.875960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.876235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:05.876235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:05.876235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:41:05.876235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:05.876235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:05.876235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.876235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:05.876235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:05.876235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:05.876235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.876235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:05.876235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:05.876235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:05.876235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:05.876235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:05.876235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:05.876235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:05.876235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:05.876584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:05.876584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:05.876584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:05.876584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:05.876584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:05.876584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.876584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:05.880091 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:05.880288 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:05.880370 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:05.880476 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:05.881417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:05.881681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.882146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:05.882146 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:05.882146 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:05.882146 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:05.882146 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:05.882146 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:05.882146 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.882146 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:05.882727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.882727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:05.882727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:05.882727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:05.882727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.882727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:05.882727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:05.882727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:05.882727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.882727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:05.882727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:05.882727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:05.882727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:05.882727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:05.882727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:05.882998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:05.882998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:05.882998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:05.882998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:05.882998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:05.882998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:05.882998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:05.882998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.882998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:05.883386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.883386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:05.883386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:05.883386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:05.883386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.883386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:05.883386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:05.883386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:05.883386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:05.883386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.883386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:05.883625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:05.883625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:05.883625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.883625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:05.883625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:05.883625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:05.883625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:05.883625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:05.883625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:05.883625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:05.883625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:05.883625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:05.883625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:05.883625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:05.883856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:05.883856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:05.883856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.883856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:05.884118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.884118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:05.884118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:05.884118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:05.884118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:05.884118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:05.884118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.884118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:05.884118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:05.884118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.884118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:05.884118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:05.884118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.884118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:05.884118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:05.884118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.884118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:05.884424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:05.884424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:05.884424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:05.884424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.884424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:05.884424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:05.884424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:05.884424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.884424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:05.884424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:05.884424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:05.884424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:05.884424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:05.884424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:05.884424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:05.884424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:05.884424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:05.884424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:05.884424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:05.884752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:05.884752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:05.884752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.884752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:05.898098 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:05.898371 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:05.898510 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:05.898714 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:05.899675 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:05.899877 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:05.900032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.900456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:05.900456 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:05.900456 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:05.900456 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:05.900456 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:05.900456 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:05.900456 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.900456 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:05.901032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.901032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:05.901032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:05.901032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:05.901032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.901032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:05.901032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:05.901032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.901032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:05.901032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:05.901032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:05.901032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.901442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:05.901442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:05.901442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:05.901442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:05.901442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:05.901442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:05.901442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:05.901442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:05.901442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:05.901442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:05.901442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:05.901442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:05.901442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:05.901847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.901847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:05.901847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.901847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:05.901847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:05.901847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:05.901847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:05.901847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:05.901847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.901847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:05.901847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:05.901847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.901847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:05.901847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:05.901847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.902282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:05.902282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:05.902282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:05.902282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:05.902282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:05.902282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.902282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:05.902282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:05.902282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:05.902282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:05.902282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:05.902282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:05.902282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:05.902282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:05.902282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:05.902282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:05.902282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:05.902282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:05.902282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:05.902933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:05.902933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:05.902933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:05.902933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:05.902933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:05.902933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:08.785172 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:08.785653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:08.785653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:08.786456 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:08.821782 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:08.821977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:08.822546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:08.822546 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:08.822546 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:08.822546 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:08.822546 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:08.822546 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:08.822546 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:08.822546 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:08.823582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:08.823582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:08.823582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:08.823582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:08.823582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:08.823582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:08.823582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:08.823582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:08.823582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:08.823582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:08.823582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:08.823582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:08.823877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:08.823877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:08.823877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:08.823877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:08.823877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:08.823877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:08.823877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:08.823877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:08.823877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:08.823877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:08.823877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:08.823877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:08.823877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:08.824087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:08.824087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:08.824697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:08.824697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:08.824697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:08.824697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:08.824697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:08.824697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:08.824697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:08.824697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:08.824697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:08.824697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:08.824697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:08.824697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:08.824697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:08.824697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:08.824697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:08.824697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:08.825021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:08.825021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:08.825021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:08.825021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:08.825021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:08.825021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:08.825021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:08.825021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:08.825021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:08.825021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:08.825021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:08.825021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:08.825021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:08.825021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:08.825021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:08.825021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:08.825021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:08.825021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:08.825021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:08.825421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:08.825421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:08.825421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:10.864146 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:10.864710 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:10.864710 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:10.865172 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:10.865333 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:10.865446 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:10.865575 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1742b4f6-e341-446e-90b3-d2893aeee352, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:10.865686 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-847f8846-d08f-4c7a-a597-500ef6e7ff78, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:10.865785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.866210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:10.866210 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:10.866210 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:10.866210 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:10.866210 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:10.866210 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:10.866210 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.866210 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:10.866828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.866828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:10.866828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:10.866828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:10.866828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.866828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:10.866828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:10.866828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.866828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:10.866828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:10.866828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:10.866828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.867060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:10.867060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:10.867060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:10.867060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:10.867060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:10.867060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:10.867060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:10.867060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:10.867060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:10.867060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:10.867060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:10.867060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:10.867060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:10.867279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.867279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:10.867527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.867527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:10.867527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:10.867527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:10.867527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:10.867527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:10.867527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.867527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:10.867527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:10.867527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.867527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:10.867527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:10.867527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.867527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:10.867527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:10.867527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.867904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:10.867904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:10.867904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.867904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:10.867904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:10.867904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:10.867904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.867904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:10.867904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:10.867904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:10.867904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:10.867904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:10.867904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:10.867904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:10.867904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:10.867904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:10.867904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:10.867904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:10.867904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:10.868397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:10.868397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.868397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:10.872243 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:10.872487 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:10.872581 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:10.872685 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:41:10.873016 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:10.873214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.873659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:10.873659 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:10.873659 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:10.873659 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:10.873659 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:10.873659 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:10.873659 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.873659 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:10.874244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.874244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:10.874244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:10.874244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:10.874244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.874244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:10.874244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:10.874244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:10.874244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.874244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:10.874244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:10.874244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:10.874244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:10.874244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:10.874244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:10.874678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:10.874678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:10.874678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:10.874678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:10.874678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:10.874678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:10.874678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:10.874678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.874678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:10.874857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.874857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:10.874857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:10.874857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:10.874857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.874857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:10.874857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:10.874857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:41:10.874857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:10.874857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:10.874857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.874857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:10.875062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:10.875062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:10.875062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.875062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:10.875062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:10.875062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:10.875062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:10.875062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:10.875062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:10.875062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:10.875062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:10.875062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:10.875062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:10.875062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:10.875318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:10.875318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:10.875318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.875318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:10.875627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.875627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:10.875627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:10.875627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:10.875627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:10.875627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:10.875627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.875627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:10.875627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:10.875627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.875627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:10.875627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:10.875627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.875627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:10.875627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:10.875627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.875901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:10.875901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:10.875901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:41:10.875901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:10.875901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:10.875901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.875901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:10.875901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:10.875901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:10.875901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.875901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:10.875901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:10.875901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:10.875901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:10.875901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:10.875901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:10.875901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:10.875901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:10.876298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:10.876298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:10.876298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:10.876298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:10.876298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:10.876298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.876298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:10.880382 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:10.880605 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:10.880708 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:10.880820 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:10.881114 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:10.881344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.881810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:10.881810 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:10.881810 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:10.881810 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:10.881810 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:10.881810 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:10.881810 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.881810 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:10.882369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.882369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:10.882369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:10.882369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:10.882369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.882369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:10.882369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:10.882369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:10.882369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.882369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:10.882369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:10.882369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:10.882369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:10.882369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:10.882369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:10.882677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:10.882677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:10.882677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:10.882677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:10.882677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:10.882677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:10.882677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:10.882677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.882677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:10.882972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.882972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:10.882972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:10.882972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:10.882972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.882972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:10.882972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:10.882972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:10.882972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:10.882972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.882972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:10.883183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:10.883183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:10.883183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.883183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:10.883183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:10.883183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:10.883183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:10.883183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:10.883183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:10.883183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:10.883183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:10.883183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:10.883183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:10.883183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:10.883444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:10.883444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:10.883444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.883444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:10.883777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.883777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:10.883777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:10.883777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:10.883777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:10.883777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:10.883777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.883777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:10.883777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:10.883777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.883777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:10.883777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:10.883777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.883777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:10.883777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:10.883777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.883777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:10.884284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:10.884284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:10.884284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:10.884284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.884284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:10.884284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:10.884284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:10.884284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.884284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:10.884284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:10.884284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:10.884284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:10.884284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:10.884284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:10.884284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:10.884284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:10.884284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:10.884284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:10.884284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:10.884815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:10.884815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:10.884815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.884815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:10.898604 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:10.898799 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:10.898922 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:10.899071 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:10.899393 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:10.899608 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:10.899776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.900208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:10.900208 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:10.900208 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:10.900208 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:10.900208 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:10.900208 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:10.900208 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.900208 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:10.900829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.900829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:10.900829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:10.900829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:10.900829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.900829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:10.900829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:10.900829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.900829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:10.900829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:10.900829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:10.900829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.901173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:10.901173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:10.901173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:10.901173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:10.901173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:10.901173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:10.901173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:10.901173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:10.901173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:10.901173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:10.901173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:10.901173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:10.901173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:10.901409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.901409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:10.901509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.901509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:10.901509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:10.901509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:10.901509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:10.901509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:10.901509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.901509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:10.901509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:10.901509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.901509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:10.901509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:10.901509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.901509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:10.901509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:10.901509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:10.901989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:10.901989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:10.901989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.901989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:10.901989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:10.901989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:10.901989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:10.901989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:10.901989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:10.901989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:10.901989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:10.901989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:10.901989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:10.901989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:10.901989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:10.901989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:10.901989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:10.901989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:10.901989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:10.902367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:10.902367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:10.902367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:13.785704 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:13.787817 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:13.787817 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:13.787817 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:13.818787 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:13.819035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:13.821126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:13.821126 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:13.821126 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:13.821126 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:13.821126 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:13.821126 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:13.821126 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:13.821126 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:13.821126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:13.821126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:13.821126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:13.821126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:13.821126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:13.821126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:13.821126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:13.821472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:13.821472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:13.821472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:13.821472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:13.821472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:13.821472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:13.821472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:13.821472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:13.821472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:13.821472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:13.821472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:13.821472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:13.821472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:13.821472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:13.821692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:13.821692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:13.821692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:13.821692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:13.821692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:13.821692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:13.821692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:13.821692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:13.821692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:13.821692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:13.821692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:13.821692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:13.821692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:13.821692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:13.821974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:13.822306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:13.822306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:13.822306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:13.822306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:13.822306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:13.822306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:13.822306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:13.822306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:13.822306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:15.864643 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:15.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:15.865156 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:15.866016 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:15.866240 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:15.866411 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:15.866561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1742b4f6-e341-446e-90b3-d2893aeee352, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:15.866705 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-847f8846-d08f-4c7a-a597-500ef6e7ff78, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:15.866809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.867315 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:15.867315 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:15.867315 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:15.867315 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:15.867315 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:15.867315 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:15.867315 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.867315 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:15.868107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.868107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:15.868107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:15.868107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:15.868107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.868107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:15.868107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:15.868107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.868107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:15.868107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:15.868107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:15.868107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.868426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:15.868426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:15.868426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:15.868426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:15.868426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:15.868426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:15.868426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:15.868426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:15.868426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:15.868426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:15.868426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:15.868426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:15.868426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:15.868727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.868727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:15.868907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.868907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:15.868907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:15.868907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:15.868907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:15.868907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:15.868907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.868907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:15.868907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:15.868907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.868907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:15.868907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:15.868907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.868907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:15.868907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:15.868907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.869207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:15.869207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:15.869207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.869207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:15.869207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:15.869207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:15.869207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.869207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:15.869207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:15.869207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:15.869207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:15.869207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:15.869207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:15.869207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:15.869207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:15.869207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:15.869207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:15.869207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:15.869207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:15.869705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:15.869705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.869705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:15.872683 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:15.872996 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:15.873116 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:15.873213 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:41:15.873609 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:15.873788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.874375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:15.874375 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:15.874375 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:15.874375 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:15.874375 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:15.874375 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:15.874375 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.874375 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:15.875104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.875104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:15.875104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:15.875104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:15.875104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.875104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:15.875104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:15.875104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:15.875104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.875104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:15.875104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:15.875104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:15.875104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:15.875104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:15.875104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:15.875400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:15.875400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:15.875400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:15.875400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:15.875400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:15.875400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:15.875400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:15.875400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.875400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:15.875992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.875992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:15.875992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:15.875992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:15.875992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.875992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:15.875992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:15.875992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:41:15.875992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:15.875992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:15.875992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.875992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:15.876389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:15.876389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:15.876389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.876389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:15.876389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:15.876389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:15.876389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:15.876389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:15.876389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:15.876389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:15.876389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:15.876389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:15.876389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:15.876389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:15.876652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:15.876652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:15.876652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.876652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:15.876981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.876981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:15.876981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:15.876981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:15.876981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:15.876981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:15.876981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.876981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:15.876981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:15.876981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.876981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:15.876981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:15.876981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.876981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:15.876981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:15.876981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:15.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:15.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:41:15.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:15.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:15.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:15.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:15.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:15.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:15.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:15.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:15.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:15.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:15.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:15.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:15.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:15.877625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:15.877625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:15.877625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:15.877625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:15.877625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:15.877625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.877625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:15.880750 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:15.880992 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:15.881099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:15.881228 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:15.881523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:15.881737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.882113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:15.882113 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:15.882113 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:15.882113 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:15.882113 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:15.882113 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:15.882113 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.882113 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:15.882853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.882853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:15.882853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:15.882853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:15.882853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.882853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:15.882853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:15.882853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:15.882853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.882853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:15.882853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:15.882853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:15.882853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:15.882853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:15.882853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:15.883119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:15.883119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:15.883119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:15.883119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:15.883119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:15.883119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:15.883119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:15.883119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.883119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:15.883725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.883725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:15.883725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:15.883725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:15.883725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.883725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:15.883725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:15.883725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:15.883725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:15.883725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.883725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:15.883964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:15.883964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:15.883964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.883964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:15.883964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:15.883964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:15.883964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:15.883964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:15.883964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:15.883964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:15.883964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:15.883964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:15.883964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:15.883964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:15.884187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:15.884187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:15.884187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.884187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:15.884756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.884756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:15.884756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:15.884756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:15.884756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:15.884756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:15.884756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.884756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:15.884756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:15.884756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.884756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:15.884756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:15.884756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.884756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:15.884756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:15.884756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.884756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:15.885116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:15.885116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:15.885116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:15.885116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.885116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:15.885116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:15.885116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:15.885116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.885116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:15.885116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:15.885116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:15.885116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:15.885116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:15.885116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:15.885116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:15.885116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:15.885116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:15.885116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:15.885116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:15.885463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:15.885463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:15.885463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.885463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:15.898984 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:15.899294 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:15.899421 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:15.899576 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:15.899877 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:15.900117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:15.900322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.900792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:15.900792 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:15.900792 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:15.900792 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:15.900792 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:15.900792 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:15.900792 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.900792 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:15.901667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.901667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:15.901667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:15.901667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:15.901667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.901667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:15.901667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:15.901667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.901667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:15.901667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:15.901667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:15.901667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.901893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:15.901893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:15.901893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:15.901893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:15.901893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:15.901893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:15.901893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:15.901893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:15.901893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:15.901893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:15.901893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:15.901893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:15.901893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:15.902095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.902095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:15.902624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.902624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:15.902624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:15.902624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:15.902624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:15.902624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:15.902624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.902624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:15.902624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:15.902624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.902624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:15.902624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:15.902624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.902624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:15.902624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:15.902624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:15.902903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:15.902903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:15.902903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.902903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:15.902903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:15.902903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:15.902903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:15.902903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:15.902903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:15.902903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:15.902903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:15.902903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:15.902903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:15.902903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:15.902903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:15.902903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:15.902903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:15.902903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:15.902903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:15.903215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:15.903215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:15.903215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:18.786539 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:18.789247 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:18.789247 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:18.789247 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:18.826859 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:18.827025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:18.828880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:18.828880 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:18.828880 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:18.828880 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:18.828880 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:18.828880 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:18.828880 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:18.828880 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:18.828880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:18.828880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:18.828880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:18.828880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:18.828880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:18.828880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:18.828880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:18.829220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:18.829220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:18.829220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:18.829220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:18.829220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:18.829220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:18.829220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:18.829220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:18.829220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:18.829220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:18.829220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:18.829220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:18.829220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:18.829220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:18.829477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:18.829477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:18.829477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:18.829477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:18.829477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:18.829477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:18.829477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:18.829477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:18.829477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:18.829477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:18.829477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:18.829477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:18.829477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:18.829477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:18.829717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:18.830036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:18.830036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:18.830036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:18.830036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:18.830036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:18.830036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:18.830036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:18.830036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:18.830036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:20.865060 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:20.865810 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:20.866043 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:20.866836 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:20.866972 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:20.867082 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:20.867212 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1742b4f6-e341-446e-90b3-d2893aeee352, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:20.867365 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-847f8846-d08f-4c7a-a597-500ef6e7ff78, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:20.867468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.868065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:20.868065 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:20.868065 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:20.868065 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:20.868065 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:20.868065 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:20.868065 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.868065 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:20.869017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.869017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:20.869017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:20.869017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:20.869017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.869017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:20.869017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:20.869017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.869017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:20.869017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:20.869017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:20.869017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.870470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:20.870470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:20.870470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:20.870470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:20.870470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:20.870470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:20.870470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:20.870470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:20.870470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:20.870470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:20.870470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:20.870470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:20.870470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:20.871086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.871086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:20.871086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.871086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:20.871086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:20.871086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:20.871086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:20.871086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:20.871086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.871086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:20.871086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:20.871086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.871086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:20.871086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:20.871086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.871814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:20.871814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:20.871814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.871814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:20.871814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:20.871814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.871814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:20.871814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:20.871814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:20.871814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.871814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:20.871814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:20.871814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:20.871814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:20.871814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:20.871814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:20.871814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:20.871814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:20.871814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:20.872709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:20.872709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:20.872709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:20.872709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:20.872709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.872709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:20.873150 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:20.873469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:20.873721 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:20.873923 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:41:20.874343 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:20.874499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.877006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:20.877006 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:20.877006 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:20.877006 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:20.877006 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:20.877006 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:20.877006 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.877006 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:20.877006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.877006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:20.877006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:20.877006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:20.877006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.877006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:20.877006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:20.877006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:20.877006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.877006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:20.877836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:20.877836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:20.877836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:20.877836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:20.877836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:20.877836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:20.877836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:20.877836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:20.877836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:20.877836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:20.877836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:20.877836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:20.877836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.877836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:20.877836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.878081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:20.878081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:20.878081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:20.878081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.878081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:20.878081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:20.878081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:41:20.878081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:20.878081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:20.878081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.878081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:20.878081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:20.878081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:20.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:20.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:20.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:20.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:20.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:20.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:20.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:20.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:20.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:20.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:20.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:20.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:20.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:20.878525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.878525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:20.878525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.878525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:20.878525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:20.878525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:20.878525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:20.878525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:20.878525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.878525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:20.878525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:20.878525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.878525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:20.878525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:20.878525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.878765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:20.878765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:20.878765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.878765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:20.878765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:20.878765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:41:20.878765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:20.878765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:20.878765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.878765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:20.878765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:20.878765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:20.878765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.878765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:20.878765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:20.878765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:20.878765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:20.878765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:20.878765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:20.879054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:20.879054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:20.879054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:20.879054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:20.879054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:20.879054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:20.879054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:20.879054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.879054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:20.881099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:20.881725 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:20.881725 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:20.881725 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:20.881927 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:20.882125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.884532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:20.884532 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:20.884532 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:20.884532 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:20.884532 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:20.884532 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:20.884532 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.884532 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:20.884532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.884532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:20.884532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:20.884532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:20.884532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.884532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:20.884532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:20.884532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:20.884532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.884532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:20.884874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:20.884874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:20.884874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:20.884874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:20.884874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:20.884874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:20.884874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:20.884874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:20.884874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:20.884874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:20.884874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:20.884874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:20.884874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.884874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:20.884874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.885235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:20.885235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:20.885235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:20.885235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.885235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:20.885235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:20.885235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:20.885235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:20.885235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.885235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:20.885235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:20.885235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:20.885235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.885461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:20.885461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:20.885461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:20.885461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:20.885461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:20.885461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:20.885461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:20.885461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:20.885461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:20.885461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:20.885461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:20.885461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:20.885461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:20.885691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.885691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:20.885691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.885691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:20.885691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:20.885691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:20.885691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:20.885691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:20.885691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.885691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:20.885691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:20.885691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.885691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:20.885691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:20.885691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.885913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:20.885913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:20.885913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.885913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:20.885913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:20.885913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:20.885913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:20.885913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.885913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:20.885913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:20.885913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:20.885913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.885913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:20.885913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:20.885913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:20.885913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:20.885913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:20.885913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:20.885913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:20.886192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:20.886192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:20.886192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:20.886192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:20.886192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:20.886192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:20.886192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.886192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:20.899404 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:20.900600 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:20.900600 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:20.900600 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:20.900853 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:20.900990 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:20.901200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.903740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:20.903740 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:20.903740 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:20.903740 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:20.903740 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:20.903740 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:20.903740 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.903740 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:20.903740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.903740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:20.903740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:20.903740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:20.903740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.903740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:20.903740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:20.904180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.904180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:20.904180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:20.904180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:20.904180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.904180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:20.904180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:20.904180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:20.904180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:20.904180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:20.904180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:20.904180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:20.904180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:20.904180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:20.904542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:20.904542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:20.904542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:20.904542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:20.904542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.904542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:20.904542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.904542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:20.904542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:20.904542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:20.904542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:20.904542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:20.904542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.904542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:20.904903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:20.905382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:20.905382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:20.905382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:20.905382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:20.905382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:20.905382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:20.905382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:20.905382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:20.905382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:20.905382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:23.787757 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:23.788351 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:23.788351 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:23.788976 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:23.823416 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:23.823416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:23.823674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:23.823674 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:23.823674 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:23.823674 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:23.823674 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:23.823674 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:23.823674 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:23.823674 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:23.824760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:23.824760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:23.824760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:23.824760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:23.824760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:23.824760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:23.824760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:23.824760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:23.824760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:23.824760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:23.824760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:23.824760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:23.825100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:23.825100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:23.825100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:23.825100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:23.825100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:23.825100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:23.825100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:23.825100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:23.825100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:23.825100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:23.825100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:23.825100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:23.825100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:23.825430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:23.825430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:23.825898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:23.825898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:23.825898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:23.825898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:23.825898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:23.825898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:23.825898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:23.825898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:23.825898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:23.825898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:23.825898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:23.825898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:23.825898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:23.825898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:23.825898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:23.825898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:23.826309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:23.826309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:23.826309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:23.826309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:23.826309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:23.826309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:23.826309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:23.826309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:23.826309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:23.826309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:23.826309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:23.826309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:23.826309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:23.826309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:23.826309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:23.826309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:23.826309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:23.826309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:23.826309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:23.826809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:23.826809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:23.826809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:25.865908 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:25.866582 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:25.866582 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:25.868345 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:25.868345 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:25.868345 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:25.868345 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1742b4f6-e341-446e-90b3-d2893aeee352, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:25.868345 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-847f8846-d08f-4c7a-a597-500ef6e7ff78, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:25.868345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.871710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:25.871710 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:25.871710 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:25.871710 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:25.871710 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:25.871710 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:25.871710 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.871710 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:25.871710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.871710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:25.871710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:25.871710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:25.871710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.871710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:25.871710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:25.872523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.872523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:25.872523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:25.872523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:25.872523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.872523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:25.872523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:25.872523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:25.872523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:25.872523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:25.872523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:25.872523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:25.872523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:25.872523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:25.873077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:25.873077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:25.873077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:25.873077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:25.873077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.873077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:25.873077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.873077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:25.873077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:25.873077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:25.873077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:25.873077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:25.873077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.873077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:25.873689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:25.874422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:25.874422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:25.874422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:25.874422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:25.874422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:25.874422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:25.874422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:25.874422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:25.874422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.874422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:25.874422 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:25.875366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:25.875605 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:25.875836 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:41:25.876465 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:25.876733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.877286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:25.877286 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:25.877286 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:25.877286 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:25.877286 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:25.877286 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:25.877286 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.877286 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:25.880293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.880293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:25.880293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:25.880293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:25.880293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.880293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:25.880293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:25.880293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:25.880293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.880293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:25.880293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:25.880293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:25.880293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:25.880293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:25.880293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:25.880978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:25.880978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:25.880978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:25.880978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:25.880978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:25.880978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:25.880978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:25.880978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.880978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:25.880978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.880978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:25.880978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:25.880978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:25.880978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.881485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:25.881485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:25.881485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:41:25.881485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:25.881485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:25.881485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.881485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:25.881485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:25.881485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:25.881485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.881485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:25.881485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:25.881485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:25.881884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:25.881884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:25.881884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:25.881884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:25.881884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:25.881884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:25.881884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:25.881884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:25.881884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:25.881884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:25.881884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.881884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:25.882339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.882339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:25.882339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:25.882339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:25.882339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:25.882339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:25.882339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.882339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:25.882339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:25.882339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.882339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:25.882339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:25.882339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.882339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:25.882339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:25.882339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.882834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:25.882834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:25.882834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:41:25.882834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:25.882834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:25.882834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.882834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:25.882834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:25.882834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:25.882834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.882834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:25.882834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:25.882834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:25.882834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:25.882834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:25.882834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:25.882834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:25.882834 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:25.883400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:25.883400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:25.883400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:25.883400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:25.883400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:25.883400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.883400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:25.883400 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:25.883400 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:25.883400 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:25.883734 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:25.883734 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:25.883734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.883734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:25.883734 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:25.883734 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:25.883734 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:25.883734 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:25.883734 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:25.883734 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.883734 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:25.884137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.884137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:25.884137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:25.884137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:25.884137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.884137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:25.884137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:25.884137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:25.884137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.884137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:25.884137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:25.884137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:25.884137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:25.884137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:25.884137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:25.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:25.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:25.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:25.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:25.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:25.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:25.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:25.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:25.884812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.884812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:25.884812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:25.884812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:25.884812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.884812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:25.884812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:25.884812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:25.884812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:25.884812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.884812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:25.885233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:25.885233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:25.885233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.885233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:25.885233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:25.885233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:25.885233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:25.885233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:25.885233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:25.885233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:25.885233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:25.885233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:25.885233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:25.885233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:25.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:25.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:25.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:25.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:25.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:25.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:25.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:25.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:25.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:25.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:25.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.886101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:25.886101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:25.886101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.886101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:25.886101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:25.886101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.886101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:25.886101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:25.886101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:25.886101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:25.886101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.886101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:25.886101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:25.886101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:25.886101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.886101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:25.886101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:25.886101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:25.886101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:25.886696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:25.886696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:25.886696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:25.886696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:25.886696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:25.886696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:25.886696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:25.886696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:25.886696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:25.886696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.886696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:25.900030 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:25.900298 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:25.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:25.900621 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:25.901041 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:25.901268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:25.901440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:25.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:25.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:25.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:25.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:25.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:25.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:25.902876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.902876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:25.902876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:25.902876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:25.902876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.902876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:25.902876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:25.902876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.902876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:25.902876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:25.902876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:25.902876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.903332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:25.903332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:25.903332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:25.903332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:25.903332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:25.903332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:25.903332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:25.903332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:25.903332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:25.903332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:25.903332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:25.903332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:25.903332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:25.903782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.903782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:25.903943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.903943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:25.903943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:25.903943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:25.903943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:25.903943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:25.903943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.903943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:25.903943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:25.903943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.903943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:25.903943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:25.903943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.903943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:25.903943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:25.903943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:25.904483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:25.904483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:25.904483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.904483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:25.904483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:25.904483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:25.904483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:25.904483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:25.904483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:25.904483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:25.904483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:25.904483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:25.904483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:25.904483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:25.904483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:25.904483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:25.904483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:25.904483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:25.904483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:25.905222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:25.905222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:25.905222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:28.788226 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:28.788755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:28.788755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:28.789332 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:28.814680 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:28.814897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:28.815425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:28.815425 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:28.815425 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:28.815425 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:28.815425 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:28.815425 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:28.815425 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:28.815425 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:28.816110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:28.816110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:28.816110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:28.816110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:28.816110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:28.816110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:28.816110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:28.816110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:28.816110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:28.816110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:28.816110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:28.816110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:28.816365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:28.816365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:28.816365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:28.816365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:28.816365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:28.816365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:28.816365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:28.816365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:28.816365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:28.816365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:28.816365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:28.816365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:28.816365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:28.816569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:28.816569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:28.816828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:28.816828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:28.816828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:28.816828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:28.816828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:28.816828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:28.816828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:28.816828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:28.816828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:28.816828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:28.816828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:28.816828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:28.816828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:28.816828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:28.816828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:28.816828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:28.817089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:28.817089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:28.817089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:28.817089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:28.817089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:28.817089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:28.817089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:28.817089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:28.817089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:28.817089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:28.817089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:28.817089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:28.817089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:28.817089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:28.817089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:28.817089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:28.817089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:28.817089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:28.817089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:28.817411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:28.817411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:28.817411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:30.866508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:30.867189 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:30.867189 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:30.868019 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:30.868244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:30.868428 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:30.868557 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1742b4f6-e341-446e-90b3-d2893aeee352, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:30.868693 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-847f8846-d08f-4c7a-a597-500ef6e7ff78, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:30.868805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.869328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:30.869328 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:30.869328 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:30.869328 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:30.869328 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:30.869328 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:30.869328 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.869328 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:30.870096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.870096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:30.870096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:30.870096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:30.870096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.870096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:30.870096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:30.870096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.870096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:30.870096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:30.870096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:30.870096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.870504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:30.870504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:30.870504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:30.870504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:30.870504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:30.870504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:30.870504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:30.870504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:30.870504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:30.870504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:30.870504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:30.870504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:30.870504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:30.870845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.870845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:30.871081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.871081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:30.871081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:30.871081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:30.871081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:30.871081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:30.871081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.871081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:30.871081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:30.871081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.871081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:30.871081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:30.871081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.871081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:30.871081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:30.871081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:30.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:30.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:30.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:30.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:30.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:30.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:30.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:30.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:30.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:30.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:30.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:30.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:30.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:30.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:30.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:30.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:30.872099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:30.872099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.872099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:30.875458 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:30.876665 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:30.876778 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:30.876886 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:41:30.877326 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:30.877576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:30.878917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.878917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:30.878917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:30.878917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:30.878917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.878917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:30.878917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:30.878917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:30.878917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.878917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:30.878917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:30.878917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:30.878917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:30.878917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:30.878917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:30.879319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:30.879319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:30.879319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:30.879319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:30.879319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:30.879319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:30.879319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:30.879319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.879319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:30.879666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.879666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:30.879666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:30.879666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:30.879666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.879666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:30.879666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:30.879666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:41:30.879666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:30.879666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:30.879666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.879666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:30.879977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:30.879977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:30.879977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.879977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:30.879977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:30.879977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:30.879977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:30.879977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:30.879977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:30.879977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:30.879977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:30.879977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:30.879977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:30.879977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:30.880358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:30.880358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:30.880358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.880358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:30.880485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.880485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:30.880485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:30.880485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:30.880485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:30.880485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:30.880485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.880485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:30.880485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:30.880485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.880485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:30.880485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:30.880485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.880485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:30.880485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:30.880485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.880888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:30.880888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:30.880888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:41:30.880888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:30.880888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:30.880888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.880888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:30.880888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:30.880888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:30.880888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.880888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:30.880888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:30.880888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:30.880888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:30.880888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:30.880888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:30.880888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:30.880888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:30.881359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:30.881359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:30.881359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:30.881359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:30.881359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:30.881359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.881359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:30.882296 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:30.882484 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:30.882573 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:30.882685 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:30.882965 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:30.883151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.883498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:30.883498 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:30.883498 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:30.883498 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:30.883498 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:30.883498 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:30.883498 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.883498 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:30.884024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.884024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:30.884024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:30.884024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:30.884024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.884024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:30.884024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:30.884024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:30.884024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.884024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:30.884024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:30.884024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:30.884024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:30.884024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:30.884024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:30.884428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:30.884428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:30.884428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:30.884428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:30.884428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:30.884428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:30.884428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:30.884428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.884428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:30.884678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.884678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:30.884678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:30.884678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:30.884678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.884678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:30.884678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:30.884678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:30.884678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:30.884678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.884678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:30.884970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:30.884970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:30.884970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.884970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:30.884970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:30.884970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:30.884970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:30.884970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:30.884970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:30.884970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:30.884970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:30.884970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:30.884970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:30.884970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:30.885342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:30.885342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:30.885342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.885342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:30.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:30.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:30.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:30.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:30.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:30.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:30.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:30.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:30.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:30.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:30.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:30.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:30.885912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:30.885912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:30.885912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:30.885912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.885912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:30.885912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:30.885912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:30.885912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.885912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:30.885912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:30.885912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:30.885912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:30.885912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:30.885912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:30.885912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:30.885912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:30.885912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:30.885912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:30.885912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:30.886396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:30.886396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:30.886396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.886396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:30.900491 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:30.900769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:30.900866 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:30.901041 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:30.901397 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:30.901659 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:30.901823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.902262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:30.902262 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:30.902262 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:30.902262 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:30.902262 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:30.902262 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:30.902262 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.902262 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:30.902967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.902967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:30.902967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:30.902967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:30.902967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.902967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:30.902967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:30.902967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.902967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:30.902967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:30.902967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:30.902967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.903354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:30.903354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:30.903354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:30.903354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:30.903354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:30.903354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:30.903354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:30.903354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:30.903354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:30.903354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:30.903354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:30.903354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:30.903354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:30.903708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.903708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:30.903826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.903826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:30.903826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:30.903826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:30.903826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:30.903826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:30.903826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.903826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:30.903826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:30.903826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.903826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:30.903826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:30.903826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.903826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:30.903826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:30.903826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:30.904287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:30.904287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:30.904287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.904287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:30.904287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:30.904287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:30.904287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:30.904287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:30.904287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:30.904287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:30.904287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:30.904287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:30.904287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:30.904287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:30.904287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:30.904287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:30.904287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:30.904287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:30.904287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:30.904792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:30.904792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:30.904792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:33.788768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:33.789350 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:33.789350 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:33.790028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:33.835542 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:33.835904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:33.836497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:33.836497 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:33.836497 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:33.836497 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:33.836497 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:33.836497 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:33.836497 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:33.836497 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:33.838440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:33.838440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:33.838440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:33.838440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:33.838440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:33.838440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:33.838440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:33.838440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:33.838440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:33.838440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:33.838440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:33.838440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:33.838783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:33.838783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:33.838783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:33.838783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:33.838783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:33.838783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:33.838783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:33.838783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:33.838783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:33.838783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:33.838783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:33.838783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:33.838783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:33.838997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:33.838997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:33.839755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:33.839755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:33.839755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:33.839755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:33.839755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:33.839755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:33.839755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:33.839755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:33.839755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:33.839755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:33.839755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:33.839755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:33.839755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:33.839755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:33.839755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:33.839755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:33.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:33.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:33.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:33.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:33.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:33.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:33.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:33.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:33.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:33.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:33.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:33.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:33.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:33.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:33.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:33.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:33.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:33.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:33.840215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:33.840724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:33.840724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:33.840724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:35.867628 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:35.868335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:35.868335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:35.868686 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:35.868830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:35.868980 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:35.869198 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1742b4f6-e341-446e-90b3-d2893aeee352, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:35.869306 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-847f8846-d08f-4c7a-a597-500ef6e7ff78, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:35.869408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.869868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:35.869868 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:35.869868 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:35.869868 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:35.869868 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:35.869868 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:35.869868 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.869868 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:35.870660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.870660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:35.870660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:35.870660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:35.870660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.870660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:35.870660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:35.870660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.870660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:35.870660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:35.870660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:35.870660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.871175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:35.871175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:35.871175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:35.871175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:35.871175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:35.871175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:35.871175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:35.871175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:35.871175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:35.871175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:35.871175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:35.871175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:35.871175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:35.871678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.871678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:35.871678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.871678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:35.871678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:35.871678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:35.871678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:35.871678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:35.871678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.871678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:35.871678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:35.871678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.871678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:35.871678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:35.871678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.872231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:35.872231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:35.872231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.872231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:35.872231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:35.872231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.872231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:35.872231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:35.872231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:35.872231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.872231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:35.872231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:35.872231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:35.872231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:35.872231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:35.872231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:35.872231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:35.872231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:35.872231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:35.872933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:35.872933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:35.872933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:35.872933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:35.872933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.872933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:35.876644 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:35.876859 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:35.877001 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:35.877127 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:41:35.877440 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:35.877640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.878057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:35.878057 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:35.878057 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:35.878057 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:35.878057 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:35.878057 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:35.878057 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.878057 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:35.878820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.878820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:35.878820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:35.878820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:35.878820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.878820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:35.878820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:35.878820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:35.878820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.878820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:35.878820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:35.878820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:35.878820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:35.878820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:35.878820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:35.879395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:35.879395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:35.879395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:35.879395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:35.879395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:35.879395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:35.879395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:35.879395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.879395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:35.879762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.879762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:35.879762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:35.879762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:35.879762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.879762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:35.879762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:35.879762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:41:35.879762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:35.879762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:35.879762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.879762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:35.880132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:35.880132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:35.880132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.880132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:35.880132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:35.880132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:35.880132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:35.880132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:35.880132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:35.880132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:35.880132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:35.880132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:35.880132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:35.880132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:35.880620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:35.880620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:35.880620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.880620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:35.880620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.880620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:35.880620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:35.880620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:35.880620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:35.880620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:35.880620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.880620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:35.880620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:35.880620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.881138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:35.881138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:35.881138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.881138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:35.881138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:35.881138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.881138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:35.881138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:35.881138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:41:35.881138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:35.881138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:35.881138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.881138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:35.881138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:35.881138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:35.881138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.881138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:35.881138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:35.881138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:35.882627 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:35.882848 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:35.882939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:35.883029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:35.883597 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:35.883859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.884356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:35.884356 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:35.884356 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:35.884356 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:35.884356 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:35.884356 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:35.884356 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.884356 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:35.884969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.884969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:35.884969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:35.884969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:35.884969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.884969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:35.884969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:35.884969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:35.884969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.884969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:35.884969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:35.884969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:35.884969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:35.884969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:35.884969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:35.885338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:35.885338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:35.885338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:35.885338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:35.885338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:35.885338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:35.885338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:35.885338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.885338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:35.885819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.885819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:35.885819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:35.885819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:35.885819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.885819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:35.885819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:35.885819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:35.885819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:35.885819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.885819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:35.886123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:35.886123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:35.886123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.886123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:35.886123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:35.886123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:35.886123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:35.886123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:35.886123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:35.886123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:35.886123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:35.886123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:35.886123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:35.886123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:35.886475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:35.886475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:35.886475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.886475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:35.886705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.886705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:35.886705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:35.886705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:35.886705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:35.886705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:35.886705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.886705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:35.886705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:35.886705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.886705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:35.886705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:35.886705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.886705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:35.886705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:35.886705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.886705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:35.887160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:35.887160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:35.887160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:35.887160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.887160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:35.887160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:35.887160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:35.887160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.887160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:35.887160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:35.887160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:35.887160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:35.887160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:35.887160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:35.887160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:35.887160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:35.887160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:35.887160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:35.887160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:35.887660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:35.887660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:35.887660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.887660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:35.900886 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:35.901128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:35.901225 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:35.901383 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:35.901854 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:35.902093 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:35.902270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.903032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:35.903032 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:35.903032 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:35.903032 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:35.903032 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:35.903032 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:35.903032 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.903032 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:35.903914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.903914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:35.903914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:35.903914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:35.903914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.903914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:35.903914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:35.903914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.903914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:35.903914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:35.903914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:35.903914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.904195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:35.904195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:35.904195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:35.904195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:35.904195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:35.904195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:35.904195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:35.904195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:35.904195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:35.904195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:35.904195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:35.904195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:35.904195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:35.904481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.904481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:35.904743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.904743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:35.904743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:35.904743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:35.904743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:35.904743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:35.904743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.904743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:35.904743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:35.904743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.904743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:35.904743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:35.904743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.904743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:35.904743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:35.904743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:35.905156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:35.905156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:35.905156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.905156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:35.905156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:35.905156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:35.905156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:35.905156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:35.905156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:35.905156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:35.905156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:35.905156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:35.905156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:35.905156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:35.905156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:35.905156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:35.905156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:35.905156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:35.905156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:35.905544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:35.905544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:35.905544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:38.789098 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:38.789542 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:38.789542 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:38.790620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:38.836241 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:38.836464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:38.836999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:38.836999 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:38.836999 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:38.836999 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:38.836999 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:38.836999 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:38.836999 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:38.836999 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:38.837679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:38.837679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:38.837679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:38.837679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:38.837679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:38.837679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:38.837679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:38.837679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:38.837679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:38.837679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:38.837679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:38.837679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:38.837932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:38.837932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:38.837932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:38.837932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:38.837932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:38.837932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:38.837932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:38.837932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:38.837932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:38.837932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:38.837932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:38.837932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:38.837932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:38.838196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:38.838196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:38.838519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:38.838519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:38.838519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:38.838519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:38.838519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:38.838519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:38.838519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:38.838519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:38.838519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:38.838519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:38.838519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:38.838519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:38.838519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:38.838519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:38.838519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:38.838519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:38.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:38.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:38.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:38.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:38.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:38.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:38.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:38.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:38.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:38.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:38.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:38.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:38.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:38.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:38.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:38.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:38.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:38.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:38.838823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:38.839130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:38.839130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:38.839130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:40.868032 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:40.868630 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:40.868630 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:40.870519 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:40.870519 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:40.870519 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:40.870519 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1742b4f6-e341-446e-90b3-d2893aeee352, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:40.870519 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-847f8846-d08f-4c7a-a597-500ef6e7ff78, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:40.870519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.870979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:40.870979 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:40.870979 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:40.870979 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:40.870979 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:40.870979 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:40.870979 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.870979 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:40.872510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.872510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:40.872510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:40.872510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:40.872510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.872510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:40.872510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:40.872510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.872510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:40.872510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:40.872510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:40.872510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.872950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:40.872950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:40.872950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:40.872950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:40.872950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:40.872950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:40.872950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:40.872950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:40.872950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:40.872950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:40.872950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:40.872950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:40.872950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:40.873396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.873396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:40.873704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.873704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:40.873704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:40.873704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:40.873704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:40.873704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:40.873704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.873704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:40.873704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:40.873704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.873704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:40.873704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:40.873704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.873704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:40.873704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:40.873704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.874344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:40.874344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:40.874344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.874344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:40.874344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:40.874344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:40.874344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.874344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:40.874344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:40.874344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:40.874344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:40.874344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:40.874344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:40.874344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:40.874344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:40.874344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:40.874344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:40.874344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:40.874344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:40.874925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:40.874925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.874925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:40.877034 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:40.877342 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:40.877471 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:40.877562 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:41:40.878492 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:40.878492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.878615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:40.878615 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:40.878615 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:40.878615 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:40.878615 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:40.878615 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:40.878615 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.878615 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:40.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:40.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:40.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:40.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:40.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:40.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:40.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:40.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:40.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:40.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:40.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:40.879115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:40.879575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:40.879575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:40.879575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:40.879575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:40.879575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:40.879575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:40.879575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:40.879575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.879575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:40.879850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.879850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:40.879850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:40.879850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:40.879850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.879850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:40.879850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:40.879850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:41:40.879850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:40.879850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:40.879850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.879850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:40.880201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:40.880201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:40.880201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.880201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:40.880201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:40.880201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:40.880201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:40.880201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:40.880201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:40.880201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:40.880201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:40.880201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:40.880201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:40.880201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:40.880628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:40.880628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:40.880628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.880628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:40.880628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.880628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:40.880628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:40.880628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:40.880628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:40.880628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:40.880628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.880628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:40.880628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:40.880628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.881065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:40.881065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:40.881065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.881065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:40.881065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:40.881065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.881065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:40.881065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:40.881065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:41:40.881065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:40.881065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:40.881065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.881065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:40.881065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:40.881065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:40.881065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.881065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:40.881065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:40.881065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:40.881603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:40.881603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:40.881603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:40.881603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:40.881603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:40.881603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:40.881603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:40.881603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:40.881603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:40.881603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:40.881603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.881603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:40.883031 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:40.883339 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:40.883470 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:40.883626 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:40.884322 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:40.884322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.884725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:40.884725 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:40.884725 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:40.884725 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:40.884725 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:40.884725 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:40.884725 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.884725 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:40.885492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.885492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:40.885492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:40.885492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:40.885492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.885492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:40.885492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:40.885492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:40.885492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.885492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:40.885492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:40.885492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:40.885492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:40.885492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:40.885492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:40.885777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:40.885777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:40.885777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:40.885777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:40.885777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:40.885777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:40.885777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:40.885777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.885777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:40.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:40.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:40.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:40.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:40.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:40.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:40.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:40.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.886340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:40.886633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:40.886633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:40.886633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.886633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:40.886633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:40.886633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:40.886633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:40.886633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:40.886633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:40.886633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:40.886633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:40.886633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:40.886633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:40.886633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:40.886888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:40.886888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:40.886888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.886888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:40.887370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.887370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:40.887370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:40.887370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:40.887370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:40.887370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:40.887370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.887370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:40.887370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:40.887370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.887370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:40.887370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:40.887370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.887370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:40.887370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:40.887370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.887370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:40.887784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:40.887784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:40.887784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:40.887784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.887784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:40.887784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:40.887784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:40.887784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.887784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:40.887784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:40.887784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:40.887784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:40.887784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:40.887784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:40.887784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:40.887784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:40.887784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:40.887784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:40.887784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:40.888167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:40.888167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:40.888167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.888167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:40.901354 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:40.901755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:40.901889 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:40.902048 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:40.902392 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:40.902643 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:40.902830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.903313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:40.903313 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:40.903313 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:40.903313 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:40.903313 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:40.903313 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:40.903313 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.903313 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:40.903987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.903987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:40.903987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:40.903987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:40.903987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.903987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:40.903987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:40.903987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.903987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:40.903987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:40.903987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:40.903987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.904245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:40.904245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:40.904245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:40.904245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:40.904245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:40.904245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:40.904245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:40.904245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:40.904245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:40.904245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:40.904245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:40.904245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:40.904245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:40.904494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.904494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:40.904774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.904774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:40.904774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:40.904774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:40.904774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:40.904774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:40.904774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.904774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:40.904774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:40.904774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.904774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:40.904774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:40.904774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.904774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:40.904774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:40.904774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:40.905048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:40.905048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:40.905048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.905048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:40.905048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:40.905048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:40.905048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:40.905048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:40.905048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:40.905048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:40.905048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:40.905048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:40.905048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:40.905048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:40.905048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:40.905048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:40.905048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:40.905048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:40.905048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:40.905397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:40.905397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:40.905397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:43.789521 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:43.789923 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:43.789961 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:43.790645 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:43.820378 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:43.820618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:43.821048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:43.821048 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:43.821048 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:43.821048 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:43.821048 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:43.821048 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:43.821048 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:43.821048 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:43.821751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:43.821751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:43.821751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:43.821751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:43.821751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:43.821751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:43.821751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:43.821751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:43.821751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:43.821751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:43.821751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:43.821751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:43.822008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:43.822008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:43.822008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:43.822008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:43.822008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:43.822008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:43.822008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:43.822008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:43.822008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:43.822008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:43.822008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:43.822008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:43.822008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:43.822203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:43.822203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:43.822482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:43.822482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:43.822482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:43.822482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:43.822482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:43.822482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:43.822482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:43.822482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:43.822482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:43.822482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:43.822482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:43.822482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:43.822482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:43.822482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:43.822482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:43.822482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:43.822884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:43.822884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:43.822884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:43.822884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:43.822884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:43.822884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:43.822884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:43.822884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:43.822884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:43.822884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:43.822884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:43.822884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:43.822884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:43.822884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:43.822884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:43.822884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:43.822884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:43.822884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:43.822884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:43.823432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:43.823432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:43.823432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:45.868498 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:45.869050 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:45.869050 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:45.869694 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:45.869877 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:45.869991 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1742b4f6-e341-446e-90b3-d2893aeee352, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:45.870097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-847f8846-d08f-4c7a-a597-500ef6e7ff78, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:45.870195 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:45.870322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.870755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:45.870755 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:45.870755 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:45.870755 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:45.870755 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:45.870755 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:45.870755 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.870755 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:45.871478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.871478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:45.871478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:45.871478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:45.871478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.871478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:45.871478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:45.871478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.871478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:45.871478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:45.871478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:45.871478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.871753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:45.871753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:45.871753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:45.871753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:45.871753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:45.871753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:45.871753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:45.871753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:45.871753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:45.871753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:45.871753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:45.871753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:45.871753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:45.872001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.872001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:45.872170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.872170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:45.872170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:45.872170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:45.872170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:45.872170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:45.872170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.872170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:45.872170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:45.872170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.872170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:45.872170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:45.872170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.872170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:45.872170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:45.872170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:45.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:45.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:45.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:45.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:45.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:45.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:45.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:45.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:45.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:45.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:45.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:45.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:45.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:45.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:45.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:45.872518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:45.872879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:45.872879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.872879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:45.877366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:45.877718 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:45.877806 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:45.877902 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:41:45.878190 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:45.878355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.878832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:45.878832 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:45.878832 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:45.878832 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:45.878832 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:45.878832 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:45.878832 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.878832 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:45.879486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.879486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:45.879486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:45.879486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:45.879486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.879486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:45.879486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:45.879486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:45.879486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.879486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:45.879486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:45.879486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:45.879486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:45.879486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:45.879486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:45.879763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:45.879763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:45.879763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:45.879763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:45.879763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:45.879763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:45.879763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:45.879763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.879763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:45.880118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.880118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:45.880118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:45.880118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:45.880118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.880118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:45.880118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:45.880118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:41:45.880118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:45.880118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:45.880118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.880118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:45.880346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:45.880346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:45.880346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.880346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:45.880346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:45.880346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:45.880346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:45.880346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:45.880346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:45.880346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:45.880346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:45.880346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:45.880346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:45.880346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:45.880588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:45.880588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:45.880588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.880588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:45.880865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.880865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:45.880865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:45.880865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:45.880865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:45.880865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:45.880865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.880865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:45.880865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:45.880865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.880865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:45.880865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:45.880865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.880865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:45.880865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:45.880865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.881119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:45.881119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:45.881119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:41:45.881119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:45.881119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:45.881119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.881119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:45.881119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:45.881119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:45.881119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.881119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:45.881119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:45.881119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:45.881119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:45.881119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:45.881119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:45.881119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:45.881119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:45.881429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:45.881429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:45.881429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:45.881429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:45.881429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:45.881429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.881429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:45.883312 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:45.883504 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:45.883591 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:45.883695 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:45.883944 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:45.884095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.884445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:45.884445 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:45.884445 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:45.884445 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:45.884445 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:45.884445 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:45.884445 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.884445 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:45.884961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.884961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:45.884961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:45.884961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:45.884961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.884961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:45.884961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:45.884961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:45.884961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.884961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:45.884961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:45.884961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:45.884961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:45.884961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:45.884961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:45.885193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:45.885193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:45.885193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:45.885193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:45.885193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:45.885193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:45.885193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:45.885193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.885193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:45.885605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.885605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:45.885605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:45.885605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:45.885605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.885605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:45.885605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:45.885605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:45.885605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:45.885605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.885605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:45.885811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:45.885811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:45.885811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.885811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:45.885811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:45.885811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:45.885811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:45.885811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:45.885811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:45.885811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:45.885811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:45.885811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:45.885811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:45.885811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:45.886027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:45.886027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:45.886027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.886027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:45.886326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.886326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:45.886326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:45.886326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:45.886326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:45.886326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:45.886326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.886326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:45.886326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:45.886326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.886326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:45.886326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:45.886326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.886326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:45.886326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:45.886326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.886326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:45.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:45.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:45.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:45.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:45.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:45.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:45.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:45.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:45.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:45.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:45.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:45.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:45.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:45.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:45.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:45.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:45.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:45.886938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:45.886938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:45.886938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.886938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:45.901771 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:45.902158 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:45.902296 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:45.905276 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:45.905276 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:45.905276 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:45.905276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.905276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:45.905276 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:45.905276 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:45.905276 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:45.905276 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:45.905276 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:45.905276 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.905276 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:45.905921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.905921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:45.905921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:45.905921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:45.905921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.905921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:45.905921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:45.905921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.905921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:45.905921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:45.905921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:45.905921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.906282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:45.906282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:45.906282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:45.906282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:45.906282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:45.906282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:45.906282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:45.906282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:45.906282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:45.906282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:45.906282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:45.906282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:45.906282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:45.906621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.906621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:45.906621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.906621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:45.906621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:45.906621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:45.906621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:45.906621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:45.906621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.906621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:45.906621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:45.906621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.906621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:45.906621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:45.906621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.907013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:45.907013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:45.907013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:45.907013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:45.907013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:45.907013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.907013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:45.907013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:45.907013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:45.907013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:45.907013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:45.907013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:45.907013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:45.907013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:45.907013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:45.907013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:45.907013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:45.907013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:45.907013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:45.907406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:45.907406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:45.907406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:45.907406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:45.907406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:45.907406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:48.790062 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:48.790894 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:48.790979 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:48.791590 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:48.819797 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:48.820046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:48.820706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:48.820706 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:48.820706 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:48.820706 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:48.820706 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:48.820706 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:48.820706 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:48.820706 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:48.821398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:48.821398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:48.821398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:48.821398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:48.821398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:48.821398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:48.821398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:48.821398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:48.821398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:48.821398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:48.821398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:48.821398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:48.821664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:48.821664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:48.821664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:48.821664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:48.821664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:48.821664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:48.821664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:48.821664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:48.821664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:48.821664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:48.821664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:48.821664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:48.821664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:48.821881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:48.821881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:48.822072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:48.822072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:48.822072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:48.822072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:48.822072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:48.822072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:48.822072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:48.822072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:48.822072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:48.822072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:48.822072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:48.822072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:48.822072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:48.822072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:48.822072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:48.822072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:48.822371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:48.822371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:48.822371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:48.822371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:48.822371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:48.822371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:48.822371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:48.822371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:48.822371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:48.822371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:48.822371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:48.822371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:48.822371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:48.822371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:48.822371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:48.822371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:48.822371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:48.822371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:48.822371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:48.822704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:48.822704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:48.822704 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:50.869519 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:50.869994 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:50.869994 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:50.870912 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:50.871106 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:50.871307 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1742b4f6-e341-446e-90b3-d2893aeee352, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:50.871455 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-847f8846-d08f-4c7a-a597-500ef6e7ff78, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:50.871610 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:50.871946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.872405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:50.872405 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:50.872405 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:50.872405 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:50.872405 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:50.872405 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:50.872405 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.872405 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:50.873371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.873371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:50.873371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:50.873371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:50.873371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.873371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:50.873371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:50.873371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.873371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:50.873371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:50.873371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:50.873371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.873609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:50.873609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:50.873609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:50.873609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:50.873609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:50.873609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:50.873609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:50.873609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:50.873609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:50.873609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:50.873609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:50.873609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:50.873609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:50.873813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.873813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:50.874430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.874430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:50.874430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:50.874430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:50.874430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:50.874430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:50.874430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.874430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:50.874430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:50.874430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.874430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:50.874430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:50.874430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.874430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:50.874430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:50.874430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.874738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:50.874738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:50.874738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.874738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:50.874738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:50.874738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:50.874738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.874738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:50.874738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:50.874738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:50.874738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:50.874738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:50.874738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:50.874738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:50.874738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:50.874738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:50.874738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:50.874738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:50.874738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:50.875021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:50.875021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.875021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:50.877655 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:50.877924 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:50.878037 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:50.878159 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:41:50.878484 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:50.878676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.879054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:50.879054 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:50.879054 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:50.879054 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:50.879054 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:50.879054 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:50.879054 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.879054 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:50.879639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.879639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:50.879639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:50.879639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:50.879639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.879639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:50.879639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:50.879639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:50.879639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.879639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:50.879639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:50.879639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:50.879639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:50.879639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:50.879639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:50.879872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:50.879872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:50.879872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:50.879872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:50.879872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:50.879872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:50.879872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:50.879872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.879872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:50.880334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.880334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:50.880334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:50.880334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:50.880334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.880334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:50.880334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:50.880334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:41:50.880334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:50.880334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:50.880334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.880334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:50.880574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:50.880574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:50.880574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.880574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:50.880574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:50.880574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:50.880574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:50.880574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:50.880574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:50.880574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:50.880574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:50.880574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:50.880574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:50.880574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:50.880817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:50.880817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:50.880817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.880817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:50.881448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.881448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:50.881448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:50.881448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:50.881448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:50.881448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:50.881448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.881448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:50.881448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:50.881448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.881448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:50.881448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:50.881448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.881448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:50.881448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:50.881448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.881721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:50.881721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:50.881721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:41:50.881721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:50.881721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:50.881721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.881721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:50.881721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:50.881721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:50.881721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.881721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:50.881721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:50.881721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:50.881721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:50.881721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:50.881721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:50.881721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:50.881721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:50.882010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:50.882010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:50.882010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:50.882010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:50.882010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:50.882010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.882010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:50.883529 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:50.883818 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:50.883929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:50.884051 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:50.884350 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:50.884536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:50.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:50.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:50.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:50.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:50.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:50.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.884937 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:50.885729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.885729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:50.885729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:50.885729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:50.885729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.885729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:50.885729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:50.885729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:50.885729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.885729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:50.885729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:50.885729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:50.885729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:50.885729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:50.885729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:50.885957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:50.885957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:50.885957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:50.885957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:50.885957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:50.885957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:50.885957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:50.885957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.885957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:50.886689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.886689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:50.886689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:50.886689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:50.886689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.886689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:50.886689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:50.886689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:50.886689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:50.886689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.886689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:50.886889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:50.886889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:50.886889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.886889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:50.886889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:50.886889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:50.886889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:50.886889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:50.886889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:50.886889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:50.886889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:50.886889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:50.886889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:50.886889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:50.887119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:50.887119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:50.887119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.887119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:50.887846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.887846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:50.887846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:50.887846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:50.887846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:50.887846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:50.887846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.887846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:50.887846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:50.887846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.887846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:50.887846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:50.887846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.887846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:50.887846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:50.887846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.887846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:50.888102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:50.888102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:50.888102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:50.888102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.888102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:50.888102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:50.888102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:50.888102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.888102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:50.888102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:50.888102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:50.888102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:50.888102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:50.888102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:50.888102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:50.888102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:50.888102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:50.888102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:50.888102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:50.888491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:50.888491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:50.888491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.888491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:50.902612 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:50.902906 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:50.903008 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:50.903187 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:50.905671 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:50.905867 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:50.905998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.907801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:50.907801 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:50.907801 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:50.907801 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:50.907801 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:50.907801 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:50.907801 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.907801 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:50.908677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.908677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:50.908677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:50.908677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:50.908677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.908677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:50.908677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:50.908677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.908677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:50.908677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:50.908677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:50.908677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.909101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:50.909101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:50.909101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:50.909101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:50.909101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:50.909101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:50.909101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:50.909101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:50.909101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:50.909101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:50.909101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:50.909101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:50.909101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:50.909432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.909432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:50.909432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.909432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:50.909432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:50.909432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:50.909432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:50.909432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:50.909432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.909432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:50.909432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:50.909432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.909432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:50.909432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:50.909432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.909901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:50.909901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:50.909901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:50.909901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:50.909901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:50.909901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.909901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:50.909901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:50.909901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:50.909901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:50.909901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:50.909901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:50.909901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:50.909901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:50.909901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:50.909901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:50.909901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:50.909901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:50.909901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:50.910557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:50.910557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:50.910557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:50.910557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:50.910557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:50.910557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:53.791014 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:53.791538 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:53.791538 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:53.792089 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:53.799480 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=573,neutron:UPDATE=1,neutron:DELETE=9 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:41:53.829411 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:53.829721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:53.830264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:53.830264 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:53.830264 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:53.830264 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:53.830264 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:53.830264 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:53.830264 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:53.830264 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:53.831142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:53.831142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:53.831142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:53.831142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:53.831142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:53.831142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:53.831142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:53.831142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:53.831142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:53.831142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:53.831142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:53.831142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:53.831525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:53.831525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:53.831525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:53.831525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:53.831525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:53.831525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:53.831525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:53.831525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:53.831525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:53.831525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:53.831525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:53.831525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:53.831525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:53.831950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:53.831950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:53.832332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:53.832332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:53.832332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:53.832332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:53.832332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:53.832332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:53.832332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:53.832332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:53.832332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:53.832332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:53.832332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:53.832332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:53.832332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:53.832332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:53.832332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:53.832332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:53.832645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:53.832645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:53.832645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:53.832645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:53.832645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:53.832645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:53.832645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:53.832645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:53.832645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:53.832645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:53.832645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:53.832645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:53.832645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:53.832645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:53.832645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:53.832645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:53.832645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:53.832645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:53.832645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:53.832940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:53.832940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:53.832940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:55.870007 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:55.870546 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:55.870546 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:55.871278 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:55.871498 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:55.871656 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-847f8846-d08f-4c7a-a597-500ef6e7ff78, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:55.871772 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3df748d4-c112-4f5e-9b9f-8fba3dd8da91, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:55.871903 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1742b4f6-e341-446e-90b3-d2893aeee352, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:55.872012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.872553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:55.872553 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:55.872553 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:55.872553 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:55.872553 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:55.872553 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:55.872553 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.872553 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:55.873303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.873303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:55.873303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:55.873303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:55.873303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.873303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:55.873303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:55.873303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.873303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:55.873303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:55.873303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:55.873303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.873596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:55.873596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:55.873596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:55.873596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:55.873596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:55.873596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:55.873596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:55.873596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:55.873596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:55.873596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:55.873596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:55.873596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:55.873596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:55.873889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.873889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:55.874017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.874017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:55.874017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:55.874017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:55.874017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:55.874017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:55.874017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.874017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:55.874017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:55.874017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.874017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:55.874017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:55.874017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.874017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:55.874017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:55.874017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:41:55.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:55.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:55.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:55.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:55.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:55.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:55.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:55.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:55.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:55.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:55.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:55.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:55.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:55.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:55.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:55.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:55.874798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:55.874798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.874798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:55.878002 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:55.878213 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:55.878320 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:55.878418 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:41:55.878733 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:55.878958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.879343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:55.879343 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:55.879343 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:55.879343 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:55.879343 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:55.879343 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:55.879343 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.879343 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:55.879873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.879873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:55.879873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:55.879873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:55.879873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.879873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:55.879873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:55.879873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:55.879873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.879873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:55.879873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:55.879873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:55.879873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:55.879873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:55.879873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:55.880171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:55.880171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:55.880171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:55.880171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:55.880171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:55.880171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:55.880171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:55.880171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.880171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:55.880433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.880433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:55.880433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:55.880433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:55.880433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.880433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:55.880433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:55.880433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:41:55.880433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:55.880433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:55.880433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.880433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:55.880675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:55.880675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:55.880675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.880675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:55.880675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:55.880675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:55.880675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:55.880675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:55.880675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:55.880675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:55.880675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:55.880675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:55.880675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:55.880675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:55.880895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:55.880895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:55.880895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.880895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:55.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:55.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:55.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:55.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:55.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:55.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:55.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:55.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:55.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:55.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:55.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:55.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.881387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:41:55.881387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:41:55.881387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:41:55.881387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:55.881387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:55.881387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.881387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:55.881387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:55.881387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:55.881387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.881387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:55.881387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:55.881387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:55.881387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:55.881387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:55.881387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:55.881387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:55.881387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:55.881703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:55.881703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:55.881703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:55.881703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:55.881703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:55.881703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.881703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:55.883859 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:55.884053 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:55.884149 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:55.884258 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:55.884544 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:55.884734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.885100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:55.885100 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:55.885100 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:55.885100 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:55.885100 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:55.885100 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:55.885100 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.885100 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:55.885611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.885611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:41:55.885611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:55.885611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:55.885611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.885611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:55.885611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:55.885611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:41:55.885611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.885611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:41:55.885611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:55.885611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:41:55.885611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:55.885611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:41:55.885611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:41:55.885875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:55.885875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:41:55.885875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:55.885875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:41:55.885875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:41:55.885875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:55.885875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:41:55.885875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.885875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:41:55.886155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.886155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:55.886155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:55.886155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:55.886155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.886155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:55.886155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:55.886155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:55.886155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:55.886155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.886155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:55.886394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:55.886394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:55.886394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.886394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:55.886394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:55.886394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:55.886394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:55.886394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:55.886394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:55.886394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:55.886394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:55.886394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:55.886394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:55.886394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:55.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:55.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:55.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.886651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:55.886887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.886887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:55.886887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:55.886887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:55.886887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:55.886887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:55.886887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.886887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:55.886887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:55.886887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.886887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:55.886887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:55.886887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.886887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:55.886887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:55.886887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.886887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:41:55.887217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:41:55.887217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:41:55.887217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:41:55.887217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.887217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:55.887217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:55.887217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:55.887217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.887217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:55.887217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:55.887217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:55.887217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:55.887217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:55.887217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:55.887217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:55.887217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:55.887217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:55.887217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:55.887217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:55.887545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:55.887545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:55.887545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.887545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:55.903007 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:55.903231 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:55.903342 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:55.903461 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:55.903793 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:55.903986 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:55.904146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.904599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:55.904599 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:55.904599 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:55.904599 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:55.904599 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:55.904599 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:55.904599 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.904599 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:55.905140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.905140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:55.905140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:55.905140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:55.905140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.905140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:55.905140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:55.905140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.905140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:55.905140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:55.905140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:55.905140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.905412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:55.905412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:55.905412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:55.905412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:55.905412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:55.905412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:55.905412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:55.905412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:55.905412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:55.905412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:55.905412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:55.905412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:55.905412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:55.905625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.905625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:55.905798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.905798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:55.905798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:55.905798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:55.905798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:55.905798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:55.905798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.905798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:55.905798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:55.905798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.905798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:55.905798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:55.905798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.905798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:55.905798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:55.905798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:55.906084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:41:55.906084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:55.906084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.906084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:55.906084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:55.906084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:55.906084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:55.906084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:55.906084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:55.906084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:55.906084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:55.906084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:55.906084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:55.906084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:55.906084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:55.906084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:55.906084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:55.906084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:55.906084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:55.906446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:55.906446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:55.906446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:41:58.791492 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:41:58.791955 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:41:58.792045 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:41:58.792743 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:41:58.826334 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:41:58.826688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:58.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:41:58.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:58.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:41:58.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:41:58.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:58.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:41:58.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:58.827321 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:41:58.828223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:58.828223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:41:58.828223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:58.828223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:41:58.828223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:58.828223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:58.828223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:58.828223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:58.828223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:58.828223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:58.828223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:41:58.828223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:58.828643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:41:58.828643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:58.828643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:41:58.828643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:58.828643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:41:58.828643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:41:58.828643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:58.828643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:41:58.828643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:58.828643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:41:58.828643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:41:58.828643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:58.828643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:41:58.829044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:58.829044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:41:58.829210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:58.829210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:41:58.829210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:41:58.829210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:41:58.829210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:41:58.829210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:41:58.829210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:58.829210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:41:58.829210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:41:58.829210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:58.829210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:41:58.829210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:41:58.829210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:41:58.829210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:41:58.829210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:41:58.829210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:41:58.829720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:41:58.829720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:41:58.829720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:58.829720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:58.829720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:41:58.829720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:41:58.829720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:41:58.829720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:41:58.829720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:41:58.829720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:41:58.829720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:41:58.829720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:41:58.829720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:41:58.829720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:41:58.829720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:41:58.829720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:41:58.829720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:41:58.829720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:41:58.829720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:41:58.830235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:41:58.830235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:41:58.830235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:00.870490 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:00.871050 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:00.871050 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:00.871834 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:00.872099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-459211a9-6ff5-45d1-9ad2-0aff99e451a4, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:00.872244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-847f8846-d08f-4c7a-a597-500ef6e7ff78, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:00.872393 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1742b4f6-e341-446e-90b3-d2893aeee352, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:00.872548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.873070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:00.873070 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:00.873070 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:00.873070 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:00.873070 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:00.873070 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:00.873070 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.873070 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:00.873846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.873846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:00.873846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:00.873846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:00.873846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.873846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:00.873846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:00.873846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.873846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:00.873846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:00.873846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:00.873846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.874071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:00.874071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:00.874071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:00.874071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:00.874071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:00.874071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:00.874071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:00.874071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:00.874071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:00.874071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:00.874071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:00.874071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:00.874071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:00.874290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.874290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:00.874641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.874641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:00.874641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:00.874641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:00.874641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:00.874641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:00.874641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.874641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:00.874641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:00.874641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.874641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:00.874641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:00.874641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.874641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:00.874641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:00.874641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.874903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:00.874903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:00.874903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.874903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:00.874903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:00.874903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:00.874903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.874903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:00.874903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:00.874903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:00.874903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:00.874903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:00.874903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:00.874903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:00.874903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:00.874903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:00.874903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:00.874903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:00.874903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:00.875199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:00.875199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.875199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:00.878361 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:00.878616 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:00.878720 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:00.878817 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:42:00.879264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:00.879461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.880021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:00.880021 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:00.880021 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:00.880021 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:00.880021 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:00.880021 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:00.880021 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.880021 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:00.880928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.880928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:00.880928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:00.880928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:00.880928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.880928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:00.880928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:00.880928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:00.880928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.880928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:00.880928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:00.880928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:00.880928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:00.880928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:00.880928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:00.881175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:00.881175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:00.881175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:00.881175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:00.881175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:00.881175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:00.881175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:00.881175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.881175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:00.881862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.881862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:00.881862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:00.881862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:00.881862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.881862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:00.881862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:00.881862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:42:00.881862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:00.881862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:00.881862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.881862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:00.882057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:00.882057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:00.882057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.882057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:00.882057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:00.882057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:00.882057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:00.882057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:00.882057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:00.882057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:00.882057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:00.882057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:00.882057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:00.882057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:00.882284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:00.882284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:00.882284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.882284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:00.883001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.883001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:00.883001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:00.883001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:00.883001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:00.883001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:00.883001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.883001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:00.883001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:00.883001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.883001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:00.883001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:00.883001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.883001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:00.883001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:00.883001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.883244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:00.883244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:00.883244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:42:00.883244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:00.883244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:00.883244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.883244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:00.883244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:00.883244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:00.883244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.883244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:00.883244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:00.883244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:00.883244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:00.883244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:00.883244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:00.883244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:00.883244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:00.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:00.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:00.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:00.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:00.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:00.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.883534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:00.884094 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:00.884366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:00.884537 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:00.884776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:00.885189 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:00.885412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.885920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:00.885920 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:00.885920 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:00.885920 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:00.885920 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:00.885920 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:00.885920 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.885920 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:00.886965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.886965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:00.886965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:00.886965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:00.886965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.886965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:00.886965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:00.886965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:00.886965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.886965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:00.886965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:00.886965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:00.886965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:00.886965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:00.886965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:00.887230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:00.887230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:00.887230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:00.887230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:00.887230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:00.887230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:00.887230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:00.887230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.887230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:00.888446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.888446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:00.888446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:00.888446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:00.888446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.888446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:00.888446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:00.888446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:00.888446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:00.888446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.888446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:00.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:00.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:00.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:00.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:00.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:00.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:00.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:00.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:00.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:00.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:00.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:00.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:00.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:00.888886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:00.888886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:00.888886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.888886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:00.890335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.890335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:00.890335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:00.890335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:00.890335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:00.890335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:00.890335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.890335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:00.890335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:00.890335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.890335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:00.890335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:00.890335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.890335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:00.890335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:00.890335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.890335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:00.890853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:00.890853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:00.890853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:00.890853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.890853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:00.890853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:00.890853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:00.890853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.890853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:00.890853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:00.890853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:00.890853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:00.890853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:00.890853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:00.890853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:00.890853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:00.890853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:00.890853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:00.890853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:00.891189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:00.891189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:00.891189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.891189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:00.903413 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:00.903688 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:00.903919 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:00.903919 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:00.904291 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:00.904513 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:00.904673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.905102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:00.905102 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:00.905102 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:00.905102 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:00.905102 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:00.905102 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:00.905102 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.905102 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:00.905787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.905787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:00.905787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:00.905787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:00.905787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.905787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:00.905787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:00.905787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.905787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:00.905787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:00.905787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:00.905787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.906002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:00.906002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:00.906002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:00.906002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:00.906002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:00.906002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:00.906002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:00.906002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:00.906002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:00.906002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:00.906002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:00.906002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:00.906002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:00.906200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.906200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:00.906499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.906499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:00.906499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:00.906499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:00.906499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:00.906499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:00.906499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.906499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:00.906499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:00.906499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.906499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:00.906499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:00.906499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.906499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:00.906499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:00.906499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:00.906766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:00.906766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:00.906766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.906766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:00.906766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:00.906766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:00.906766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:00.906766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:00.906766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:00.906766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:00.906766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:00.906766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:00.906766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:00.906766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:00.906766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:00.906766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:00.906766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:00.906766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:00.906766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:00.907050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:00.907050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:00.907050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:03.792442 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:03.793805 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:03.793805 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:03.793981 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:03.823105 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:03.823356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:03.823960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:03.823960 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:03.823960 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:03.823960 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:03.823960 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:03.823960 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:03.823960 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:03.823960 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:03.824883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:03.824883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:03.824883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:03.824883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:03.824883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:03.824883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:03.824883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:03.824883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:03.824883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:03.824883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:03.824883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:03.824883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:03.825242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:03.825242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:03.825242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:03.825242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:03.825242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:03.825242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:03.825242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:03.825242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:03.825242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:03.825242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:03.825242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:03.825242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:03.825242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:03.825504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:03.825504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:03.826149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:03.826149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:03.826149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:03.826149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:03.826149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:03.826149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:03.826149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:03.826149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:03.826149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:03.826149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:03.826149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:03.826149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:03.826149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:03.826149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:03.826149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:03.826149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:03.826614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:03.826614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:03.826614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:03.826614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:03.826614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:03.826614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:03.826614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:03.826614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:03.826614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:03.826614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:03.826614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:03.826614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:03.826614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:03.826614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:03.826614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:03.826614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:03.826614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:03.826614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:03.826614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:03.826928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:03.826928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:03.826928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:05.871641 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:05.872121 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:05.872121 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:05.872601 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:05.872779 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-847f8846-d08f-4c7a-a597-500ef6e7ff78, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:05.872956 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1742b4f6-e341-446e-90b3-d2893aeee352, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:05.873026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.873516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:05.873516 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:05.873516 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:05.873516 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:05.873516 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:05.873516 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:05.873516 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.873516 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:05.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:05.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:05.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:05.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:05.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:05.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:05.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:05.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:05.874219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.874626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:05.874626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:05.874626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:05.874626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:05.874626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:05.874626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:05.874626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:05.874626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:05.874626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:05.874626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:05.874626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:05.874626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:05.874626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:05.875013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.875013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:05.875013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.875013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:05.875013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:05.875013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:05.875013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:05.875013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:05.875013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.875013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:05.875013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:05.875013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.875013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:05.875013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:05.875013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.875596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:05.875596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:05.875596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.875596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:05.875596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:05.875596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.875596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:05.875596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:05.875596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:05.875596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.875596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:05.875596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:05.875596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:05.875596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:05.875596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:05.875596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:05.875596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:05.875596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:05.875596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:05.876140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:05.876140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:05.876140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:05.876140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:05.876140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.876140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:05.878668 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:05.878894 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:05.878988 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:05.879079 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:42:05.879386 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:05.879729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.879972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:05.879972 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:05.879972 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:05.879972 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:05.879972 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:05.879972 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:05.879972 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.879972 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:05.880492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.880492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:05.880492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:05.880492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:05.880492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.880492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:05.880492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:05.880492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:05.880492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.880492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:05.880492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:05.880492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:05.880492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:05.880492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:05.880492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:05.880808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:05.880808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:05.880808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:05.880808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:05.880808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:05.880808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:05.880808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:05.880808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.880808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:05.881345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.881345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:05.881345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:05.881345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:05.881345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.881345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:05.881345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:05.881345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:42:05.881345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:05.881345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:05.881345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.881345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:05.881585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:05.881585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:05.881585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.881585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:05.881585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:05.881585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:05.881585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:05.881585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:05.881585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:05.881585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:05.881585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:05.881585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:05.881585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:05.881585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:05.881799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:05.881799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:05.881799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.881799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:05.882246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.882246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:05.882246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:05.882246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:05.882246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:05.882246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:05.882246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.882246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:05.882246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:05.882246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.882246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:05.882246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:05.882246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.882246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:05.882246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:05.882246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.882534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:05.882534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:05.882534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:42:05.882534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:05.882534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:05.882534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.882534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:05.882534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:05.882534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:05.882534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.882534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:05.882534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:05.882534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:05.882534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:05.882534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:05.882534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:05.882534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:05.882534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:05.882899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:05.882899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:05.882899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:05.882899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:05.882899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:05.882899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.882899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:05.884420 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:05.884762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:05.884825 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:05.884945 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:05.885241 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:05.885640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.885879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:05.885879 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:05.885879 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:05.885879 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:05.885879 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:05.885879 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:05.885879 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.885879 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:05.887302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.887302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:05.887302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:05.887302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:05.887302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.887302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:05.887302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:05.887302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:05.887302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.887302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:05.887302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:05.887302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:05.887302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:05.887302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:05.887302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:05.887727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:05.887727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:05.887727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:05.887727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:05.887727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:05.887727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:05.887727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:05.887727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.887727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:05.888686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.888686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:05.888686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:05.888686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:05.888686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.888686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:05.888686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:05.888686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:05.888686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:05.888686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.888686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:05.888969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:05.888969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:05.888969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.888969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:05.888969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:05.888969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:05.888969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:05.888969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:05.888969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:05.888969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:05.888969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:05.888969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:05.888969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:05.888969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:05.889220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:05.889220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:05.889220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.889220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:05.890377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.890377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:05.890377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:05.890377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:05.890377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:05.890377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:05.890377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.890377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:05.890377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:05.890377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.890377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:05.890377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:05.890377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.890377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:05.890377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:05.890377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.890377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:05.891128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:05.891128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:05.891128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:05.891128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.891128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:05.891128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:05.891128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:05.891128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.891128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:05.891128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:05.891128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:05.891128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:05.891128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:05.891128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:05.891128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:05.891128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:05.891128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:05.891128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:05.891128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:05.891464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:05.891464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:05.891464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.891464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:05.903834 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:05.904162 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:05.904335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:05.904524 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:05.904822 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:05.905018 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:05.905181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.905708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:05.905708 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:05.905708 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:05.905708 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:05.905708 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:05.905708 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:05.905708 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.905708 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:05.906640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.906640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:05.906640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:05.906640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:05.906640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.906640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:05.906640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:05.906640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.906640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:05.906640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:05.906640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:05.906640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.906858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:05.906858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:05.906858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:05.906858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:05.906858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:05.906858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:05.906858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:05.906858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:05.906858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:05.906858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:05.906858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:05.906858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:05.906858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:05.907114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.907114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:05.907694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.907694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:05.907694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:05.907694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:05.907694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:05.907694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:05.907694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.907694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:05.907694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:05.907694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.907694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:05.907694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:05.907694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.907694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:05.907694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:05.907694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:05.907969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:05.907969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:05.907969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.907969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:05.907969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:05.907969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:05.907969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:05.907969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:05.907969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:05.907969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:05.907969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:05.907969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:05.907969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:05.907969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:05.907969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:05.907969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:05.907969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:05.907969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:05.907969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:05.908286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:05.908286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:05.908286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:08.792874 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:08.793603 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:08.793603 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:08.793702 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:08.819139 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:08.819139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:08.819301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:08.819301 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:08.819301 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:08.819301 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:08.819301 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:08.819301 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:08.819301 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:08.819301 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:08.819815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:08.819815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:08.819815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:08.819815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:08.819815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:08.819815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:08.819815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:08.819815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:08.819815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:08.819815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:08.819815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:08.819815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:08.820114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:08.820114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:08.820114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:08.820114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:08.820114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:08.820114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:08.820114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:08.820114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:08.820114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:08.820114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:08.820114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:08.820114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:08.820114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:08.820374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:08.820374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:08.820516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:08.820516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:08.820516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:08.820516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:08.820516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:08.820516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:08.820516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:08.820516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:08.820516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:08.820516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:08.820516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:08.820516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:08.820516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:08.820516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:08.820516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:08.820516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:08.820906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:08.820906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:08.820906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:08.820906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:08.820906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:08.820906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:08.820906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:08.820906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:08.820906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:08.820906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:08.820906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:08.820906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:08.820906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:08.820906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:08.820906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:08.820906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:08.820906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:08.820906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:08.820906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:08.821231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:08.821231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:08.821231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:10.872098 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:10.872651 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:10.872651 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:10.873153 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:10.873307 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-847f8846-d08f-4c7a-a597-500ef6e7ff78, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:10.873416 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1742b4f6-e341-446e-90b3-d2893aeee352, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:10.873516 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:10.873623 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:10.873717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.874483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:10.874483 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:10.874483 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:10.874483 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:10.874483 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:10.874483 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:10.874483 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.874483 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:10.875674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.875674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:10.875674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:10.875674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:10.875674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.875674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:10.875674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:10.875674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.875674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:10.875674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:10.875674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:10.875674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.875952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:10.875952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:10.875952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:10.875952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:10.875952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:10.875952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:10.875952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:10.875952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:10.875952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:10.875952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:10.875952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:10.875952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:10.875952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:10.876176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.876176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:10.876429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.876429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:10.876429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:10.876429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:10.876429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:10.876429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:10.876429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.876429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:10.876429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:10.876429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.876429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:10.876429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:10.876429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.876429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:10.876429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:10.876429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.876711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:10.876711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:10.876711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.876711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:10.876711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:10.876711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:10.876711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.876711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:10.876711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:10.876711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:10.876711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:10.876711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:10.876711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:10.876711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:10.876711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:10.876711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:10.876711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:10.876711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:10.876711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:10.877057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:10.877057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.877057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:10.879129 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:10.879321 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:10.879413 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:10.879527 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:42:10.879767 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:10.879903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.880230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:10.880230 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:10.880230 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:10.880230 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:10.880230 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:10.880230 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:10.880230 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.880230 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:10.880892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.880892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:10.880892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:10.880892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:10.880892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.880892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:10.880892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:10.880892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:10.880892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.880892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:10.880892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:10.880892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:10.880892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:10.880892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:10.880892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:10.881307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:10.881307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:10.881307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:10.881307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:10.881307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:10.881307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:10.881307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:10.881307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.881307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:10.882047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.882047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:10.882047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:10.882047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:10.882047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.882047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:10.882047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:10.882047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:42:10.882047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:10.882047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:10.882047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.882047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:10.882312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:10.882312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:10.882312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.882312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:10.882312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:10.882312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:10.882312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:10.882312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:10.882312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:10.882312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:10.882312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:10.882312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:10.882312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:10.882312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:10.882619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:10.882619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:10.882619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.882619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:10.882952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.882952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:10.882952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:10.882952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:10.882952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:10.882952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:10.882952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.882952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:10.882952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:10.882952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.882952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:10.882952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:10.882952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.882952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:10.882952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:10.882952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.883377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:10.883377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:10.883377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:42:10.883377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:10.883377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:10.883377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.883377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:10.883377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:10.883377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:10.883377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.883377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:10.883377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:10.883377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:10.883377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:10.883377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:10.883377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:10.883377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:10.883377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:10.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:10.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:10.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:10.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:10.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:10.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.883751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:10.884882 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:10.885045 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:10.885130 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:10.885233 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:10.885481 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:10.885729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.886039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:10.886039 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:10.886039 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:10.886039 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:10.886039 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:10.886039 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:10.886039 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.886039 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:10.886541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.886541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:10.886541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:10.886541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:10.886541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.886541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:10.886541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:10.886541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:10.886541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.886541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:10.886541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:10.886541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:10.886541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:10.886541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:10.886541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:10.886800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:10.886800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:10.886800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:10.886800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:10.886800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:10.886800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:10.886800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:10.886800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.886800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:10.887110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.887110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:10.887110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:10.887110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:10.887110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.887110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:10.887110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:10.887110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:10.887110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:10.887110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.887110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:10.887354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:10.887354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:10.887354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.887354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:10.887354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:10.887354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:10.887354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:10.887354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:10.887354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:10.887354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:10.887354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:10.887354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:10.887354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:10.887354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:10.887605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:10.887605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:10.887605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.887605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:10.887820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.887820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:10.887820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:10.887820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:10.887820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:10.887820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:10.887820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.887820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:10.887820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:10.887820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.887820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:10.887820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:10.887820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.887820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:10.887820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:10.887820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.887820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:10.888452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:10.888452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:10.888452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:10.888452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.888452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:10.888452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:10.888452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:10.888452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.888452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:10.888452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:10.888452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:10.888452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:10.888452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:10.888452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:10.888452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:10.888452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:10.888452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:10.888452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:10.888452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:10.889006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:10.889006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:10.889006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.889006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:10.904357 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:10.904590 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:10.904680 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:10.904802 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:10.905051 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:10.905242 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:10.905422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.905788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:10.905788 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:10.905788 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:10.905788 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:10.905788 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:10.905788 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:10.905788 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.905788 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:10.906394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.906394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:10.906394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:10.906394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:10.906394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.906394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:10.906394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:10.906394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.906394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:10.906394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:10.906394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:10.906394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.906673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:10.906673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:10.906673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:10.906673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:10.906673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:10.906673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:10.906673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:10.906673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:10.906673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:10.906673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:10.906673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:10.906673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:10.906673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:10.906866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.906866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:10.907027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.907027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:10.907027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:10.907027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:10.907027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:10.907027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:10.907027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.907027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:10.907027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:10.907027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.907027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:10.907027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:10.907027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.907027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:10.907027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:10.907027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:10.907420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:10.907420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:10.907420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.907420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:10.907420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:10.907420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:10.907420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:10.907420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:10.907420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:10.907420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:10.907420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:10.907420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:10.907420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:10.907420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:10.907420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:10.907420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:10.907420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:10.907420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:10.907420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:10.907958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:10.907958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:10.907958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:13.793424 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:13.794608 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:13.794723 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:13.795386 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:13.847509 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:13.847763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:13.848698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:13.848698 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:13.848698 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:13.848698 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:13.848698 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:13.848698 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:13.848698 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:13.848698 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:13.853224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.054 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:13.853224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:13.853224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:13.853224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:13.853224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:13.853224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:13.853224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:13.853224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:13.853224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:13.853224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:13.853224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:13.853224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:13.853487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:13.853487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:13.853487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:13.853487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:13.853487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:13.853487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:13.853487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:13.853487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:13.853487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:13.853487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:13.853487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:13.853487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:13.853487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:13.853714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:13.853714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:13.856662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:13.856662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:13.856662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:13.856662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:13.856662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:13.856662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:13.856662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:13.856662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:13.856662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:13.856662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:13.856662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:13.856662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:13.856662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:13.856662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:13.856662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:13.856662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:13.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:13.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:13.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:13.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:13.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:13.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:13.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:13.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:13.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:13.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:13.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:13.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:13.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:13.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:13.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:13.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:13.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:13.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:13.857103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:13.857511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:13.857511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:13.857511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:15.872462 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:15.872915 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:15.872952 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:15.873707 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:15.873831 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-847f8846-d08f-4c7a-a597-500ef6e7ff78, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:15.873932 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1742b4f6-e341-446e-90b3-d2893aeee352, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:15.874037 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:15.874134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:15.874240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.874664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:15.874664 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:15.874664 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:15.874664 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:15.874664 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:15.874664 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:15.874664 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.874664 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:15.875337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.875337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:15.875337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:15.875337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:15.875337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.875337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:15.875337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:15.875337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.875337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:15.875337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:15.875337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:15.875337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.875645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:15.875645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:15.875645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:15.875645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:15.875645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:15.875645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:15.875645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:15.875645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:15.875645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:15.875645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:15.875645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:15.875645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:15.875645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:15.875891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.875891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:15.876293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.876293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:15.876293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:15.876293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:15.876293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:15.876293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:15.876293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.876293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:15.876293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:15.876293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.876293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:15.876293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:15.876293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.876293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:15.876293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:15.876293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.876628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:15.876628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:15.876628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.876628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:15.876628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:15.876628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:15.876628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.876628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:15.876628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:15.876628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:15.876628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:15.876628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:15.876628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:15.876628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:15.876628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:15.876628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:15.876628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:15.876628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:15.876628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:15.876959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:15.876959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.876959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:15.879423 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:15.879673 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:15.879761 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:15.879851 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:42:15.880104 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:15.880261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.880733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:15.880733 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:15.880733 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:15.880733 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:15.880733 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:15.880733 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:15.880733 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.880733 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:15.881228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.881228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:15.881228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:15.881228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:15.881228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.881228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:15.881228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:15.881228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:15.881228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.881228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:15.881228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:15.881228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:15.881228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:15.881228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:15.881228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:15.881533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:15.881533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:15.881533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:15.881533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:15.881533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:15.881533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:15.881533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:15.881533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.881533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:15.881821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.881821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:15.881821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:15.881821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:15.881821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.881821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:15.881821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:15.881821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:42:15.881821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:15.881821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:15.881821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.881821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:15.882037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:15.882037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:15.882037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.882037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:15.882037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:15.882037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:15.882037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:15.882037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:15.882037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:15.882037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:15.882037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:15.882037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:15.882037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:15.882037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:15.882297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:15.882297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:15.882297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.882297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:15.882521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.882521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:15.882521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:15.882521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:15.882521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:15.882521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:15.882521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.882521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:15.882521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:15.882521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.882521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:15.882521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:15.882521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.882521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:15.882521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:15.882521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.882840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:15.882840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:15.882840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:42:15.882840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:15.882840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:15.882840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.882840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:15.882840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:15.882840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:15.882840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.882840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:15.882840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:15.882840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:15.882840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:15.882840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:15.882840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:15.882840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:15.882840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:15.883170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:15.883170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:15.883170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:15.883170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:15.883170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:15.883170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.883170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:15.885139 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:15.885358 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:15.885444 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:15.885546 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:15.885777 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:15.885969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.886295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:15.886295 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:15.886295 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:15.886295 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:15.886295 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:15.886295 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:15.886295 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.886295 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:15.886799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.886799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:15.886799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:15.886799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:15.886799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.886799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:15.886799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:15.886799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:15.886799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.886799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:15.886799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:15.886799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:15.886799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:15.886799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:15.886799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:15.887039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:15.887039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:15.887039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:15.887039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:15.887039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:15.887039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:15.887039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:15.887039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.887039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:15.887379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.887379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:15.887379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:15.887379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:15.887379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.887379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:15.887379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:15.887379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:15.887379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:15.887379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.887379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:15.887615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:15.887615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:15.887615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.887615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:15.887615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:15.887615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:15.887615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:15.887615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:15.887615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:15.887615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:15.887615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:15.887615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:15.887615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:15.887615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:15.887878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:15.887878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:15.887878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.887878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:15.888165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.888165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:15.888165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:15.888165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:15.888165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:15.888165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:15.888165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.888165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:15.888165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:15.888165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.888165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:15.888165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:15.888165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.888165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:15.888165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:15.888165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.888165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:15.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:15.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:15.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:15.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:15.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:15.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:15.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:15.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:15.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:15.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:15.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:15.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:15.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:15.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:15.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:15.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:15.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:15.888803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:15.888803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:15.888803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.888803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:15.904675 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:15.905054 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:15.905196 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:15.905423 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:15.905766 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:15.906029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:15.906259 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.906846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:15.906846 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:15.906846 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:15.906846 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:15.906846 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:15.906846 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:15.906846 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.906846 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:15.907784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.907784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:15.907784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:15.907784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:15.907784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.907784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:15.907784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:15.907784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.907784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:15.907784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:15.907784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:15.907784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.908087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:15.908087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:15.908087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:15.908087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:15.908087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:15.908087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:15.908087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:15.908087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:15.908087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:15.908087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:15.908087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:15.908087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:15.908087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:15.908307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.908307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:15.908928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.908928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:15.908928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:15.908928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:15.908928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:15.908928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:15.908928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.908928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:15.908928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:15.908928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.908928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:15.908928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:15.908928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.908928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:15.908928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:15.908928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:15.909394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:15.909394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:15.909394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.909394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:15.909394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:15.909394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:15.909394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:15.909394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:15.909394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:15.909394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:15.909394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:15.909394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:15.909394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:15.909394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:15.909394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:15.909394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:15.909394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:15.909394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:15.909394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:15.909982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:15.909982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:15.909982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:18.794627 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:18.795177 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:18.795177 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:18.795661 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:18.824864 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:18.824864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:18.824994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:18.824994 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:18.824994 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:18.824994 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:18.824994 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:18.824994 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:18.824994 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:18.824994 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:18.825626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:18.825626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:18.825626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:18.825626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:18.825626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:18.825626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:18.825626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:18.825626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:18.825626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:18.825626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:18.825626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:18.825626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:18.825882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:18.825882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:18.825882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:18.825882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:18.825882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:18.825882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:18.825882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:18.825882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:18.825882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:18.825882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:18.825882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:18.825882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:18.825882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:18.826093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:18.826093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:18.826462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:18.826462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:18.826462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:18.826462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:18.826462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:18.826462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:18.826462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:18.826462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:18.826462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:18.826462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:18.826462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:18.826462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:18.826462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:18.826462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:18.826462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:18.826462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:18.826754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:18.826754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:18.826754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:18.826754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:18.826754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:18.826754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:18.826754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:18.826754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:18.826754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:18.826754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:18.826754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:18.826754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:18.826754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:18.826754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:18.826754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:18.826754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:18.826754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:18.826754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:18.826754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:18.827068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:18.827068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:18.827068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:20.872848 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:20.873347 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:20.873537 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:20.875344 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:20.875344 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-847f8846-d08f-4c7a-a597-500ef6e7ff78, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:20.875344 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:20.875344 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:20.875344 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1742b4f6-e341-446e-90b3-d2893aeee352, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:20.875344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.878285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:20.878285 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:20.878285 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:20.878285 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:20.878285 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:20.878285 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:20.878285 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.878285 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:20.878285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.878285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:20.878285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:20.878285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:20.878285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.878285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:20.878285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:20.878646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.878646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:20.878646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:20.878646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:20.878646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.878646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:20.878646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:20.878646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:20.878646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:20.878646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:20.878646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:20.878646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:20.878646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:20.878646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:20.878879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:20.878879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:20.878879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:20.878879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:20.878879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.878879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:20.878879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.878879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:20.878879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:20.878879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:20.878879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:20.878879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:20.878879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.878879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:20.879095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:20.879423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:20.879423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:20.879423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:20.879423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:20.879423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:20.879423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:20.879423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:20.879423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:20.879423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.879423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:20.879748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:20.879748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:20.879748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:20.879748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:42:20.880511 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:20.880692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.881028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:20.881028 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:20.881028 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:20.881028 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:20.881028 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:20.881028 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:20.881028 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.881028 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:20.881538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.881538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:20.881538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:20.881538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:20.881538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.881538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:20.881538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:20.881538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:20.881538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.881538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:20.881538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:20.881538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:20.881538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:20.881538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:20.881538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:20.881813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:20.881813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:20.881813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:20.881813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:20.881813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:20.881813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:20.881813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:20.881813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.881813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:20.882267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.882267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:20.882267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:20.882267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:20.882267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.882267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:20.882267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:20.882267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:42:20.882267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:20.882267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:20.882267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.882267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:20.882758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:20.882758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:20.882758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.882758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:20.882758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:20.882758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:20.882758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:20.882758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:20.882758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:20.882758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:20.882758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:20.882758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:20.882758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:20.882758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:20.883077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:20.883077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:20.883077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.883077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:20.883077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.883077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:20.883077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:20.883077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:20.883077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:20.883077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:20.883077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.883077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:20.883077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:20.883077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.883301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:20.883301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:20.883301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.883301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:20.883301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:20.883301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.883301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:20.883301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:20.883301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:42:20.883301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:20.883301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:20.883301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.883301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:20.883301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:20.883301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:20.883301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.883301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:20.883301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:20.883301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:20.883600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:20.883600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:20.883600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:20.883600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:20.883600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:20.883600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:20.883600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:20.883600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:20.883600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:20.883600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:20.883600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.883600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:20.885397 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:20.885632 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:20.885767 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:20.885889 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:20.886155 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:20.886422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.886741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:20.886741 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:20.886741 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:20.886741 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:20.886741 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:20.886741 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:20.886741 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.886741 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:20.887536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.887536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:20.887536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:20.887536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:20.887536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.887536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:20.887536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:20.887536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:20.887536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.887536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:20.887536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:20.887536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:20.887536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:20.887536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:20.887536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:20.887786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:20.887786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:20.887786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:20.887786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:20.887786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:20.887786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:20.887786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:20.887786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.887786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:20.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:20.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:20.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:20.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:20.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:20.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:20.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:20.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.888474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:20.888711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:20.888711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:20.888711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.888711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:20.888711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:20.888711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:20.888711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:20.888711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:20.888711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:20.888711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:20.888711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:20.888711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:20.888711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:20.888711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:20.888929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:20.888929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:20.888929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.888929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:20.889649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.889649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:20.889649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:20.889649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:20.889649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:20.889649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:20.889649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.889649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:20.889649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:20.889649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.889649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:20.889649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:20.889649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.889649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:20.889649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:20.889649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.889649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:20.889995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:20.889995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:20.889995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:20.889995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.889995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:20.889995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:20.889995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:20.889995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.889995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:20.889995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:20.889995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:20.889995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:20.889995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:20.889995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:20.889995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:20.889995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:20.889995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:20.889995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:20.889995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:20.890312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:20.890312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:20.890312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.890312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:20.905808 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:20.906117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:20.906260 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:20.906477 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:20.907304 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:20.907360 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:20.907530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.907994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:20.907994 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:20.907994 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:20.907994 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:20.907994 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:20.907994 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:20.907994 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.907994 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:20.908927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.908927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:20.908927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:20.908927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:20.908927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.908927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:20.908927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:20.908927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.908927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:20.908927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:20.908927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:20.908927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.909177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:20.909177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:20.909177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:20.909177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:20.909177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:20.909177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:20.909177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:20.909177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:20.909177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:20.909177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:20.909177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:20.909177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:20.909177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:20.911037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.911037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:20.911037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.911037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:20.911037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:20.911037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:20.911037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:20.911037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:20.911037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.911037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:20.911037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:20.911037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.911037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:20.911037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:20.911037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.911321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:20.911321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:20.911321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:20.911321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:20.911321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:20.911321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.911321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:20.911321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:20.911321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:20.911321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:20.911321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:20.911321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:20.911321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:20.911321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:20.911321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:20.911321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:20.911321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:20.911321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:20.911321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:20.911660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:20.911660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:20.911660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:20.911660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:20.911660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:20.911660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:23.795067 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:23.795528 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:23.795528 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:23.796238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:23.829005 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:23.829243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:23.829873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:23.829873 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:23.829873 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:23.829873 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:23.829873 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:23.829873 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:23.829873 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:23.829873 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:23.830541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:23.830541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:23.830541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:23.830541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:23.830541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:23.830541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:23.830541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:23.830541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:23.830541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:23.830541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:23.830541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:23.830541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:23.830820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:23.830820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:23.830820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:23.830820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:23.830820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:23.830820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:23.830820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:23.830820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:23.830820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:23.830820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:23.830820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:23.830820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:23.830820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:23.831106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:23.831106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:23.831275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:23.831275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:23.831275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:23.831275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:23.831275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:23.831275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:23.831275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:23.831275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:23.831275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:23.831275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:23.831275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:23.831275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:23.831275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:23.831275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:23.831275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:23.831275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:23.831554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:23.831554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:23.831554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:23.831554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:23.831554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:23.831554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:23.831554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:23.831554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:23.831554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:23.831554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:23.831554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:23.831554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:23.831554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:23.831554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:23.831554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:23.831554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:23.831554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:23.831554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:23.831554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:23.831915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:23.831915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:23.831915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:25.873344 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:25.873900 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:25.873900 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:25.874704 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:25.874962 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-847f8846-d08f-4c7a-a597-500ef6e7ff78, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:25.875152 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:25.875336 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:25.875494 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1742b4f6-e341-446e-90b3-d2893aeee352, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:25.875664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.876277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:25.876277 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:25.876277 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:25.876277 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:25.876277 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:25.876277 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:25.876277 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.876277 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:25.877044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.877044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:25.877044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:25.877044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:25.877044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.877044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:25.877044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:25.877044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.877044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:25.877044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:25.877044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:25.877044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:25.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:25.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:25.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:25.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:25.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:25.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:25.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:25.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:25.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:25.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:25.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:25.877295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:25.877502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.877502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:25.877872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.877872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:25.877872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:25.877872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:25.877872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:25.877872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:25.877872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.877872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:25.877872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:25.877872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.877872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:25.877872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:25.877872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.877872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:25.877872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:25.877872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.878176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:25.878176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:25.878176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.878176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:25.878176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:25.878176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:25.878176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.878176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:25.878176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:25.878176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:25.878176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:25.878176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:25.878176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:25.878176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:25.878176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:25.878176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:25.878176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:25.878176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:25.878176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:25.878476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:25.878476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.878476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:25.880104 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:25.880271 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:25.880363 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:25.880499 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:42:25.881021 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:25.881272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.881850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:25.881850 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:25.881850 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:25.881850 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:25.881850 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:25.881850 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:25.881850 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.881850 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:25.882546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.882546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:25.882546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:25.882546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:25.882546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.882546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:25.882546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:25.882546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:25.882546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.882546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:25.882546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:25.882546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:25.882546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:25.882546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:25.882546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:25.882879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:25.882879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:25.882879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:25.882879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:25.882879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:25.882879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:25.882879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:25.882879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.882879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:25.883393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.883393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:25.883393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:25.883393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:25.883393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.883393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:25.883393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:25.883393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:42:25.883393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:25.883393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:25.883393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.883393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:25.883610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:25.883610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:25.883610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.883610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:25.883610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:25.883610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:25.883610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:25.883610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:25.883610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:25.883610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:25.883610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:25.883610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:25.883610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:25.883610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:25.883829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:25.883829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:25.883829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.883829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:25.884366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.884366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:25.884366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:25.884366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:25.884366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:25.884366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:25.884366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.884366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:25.884366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:25.884366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.884366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:25.884366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:25.884366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.884366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:25.884366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:25.884366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.884632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:25.884632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:25.884632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:42:25.884632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:25.884632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:25.884632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.884632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:25.884632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:25.884632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:25.884632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.884632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:25.884632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:25.884632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:25.884632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:25.884632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:25.884632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:25.884632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:25.884632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:25.884982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:25.884982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:25.884982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:25.884982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:25.884982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:25.884982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.884982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:25.885676 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:25.885877 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:25.885992 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:25.886131 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:25.886726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:25.887003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.887611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:25.887611 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:25.887611 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:25.887611 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:25.887611 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:25.887611 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:25.887611 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.887611 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:25.888248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.888248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:25.888248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:25.888248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:25.888248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.888248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:25.888248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:25.888248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:25.888248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.888248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:25.888248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:25.888248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:25.888248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:25.888248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:25.888248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:25.888529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:25.888529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:25.888529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:25.888529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:25.888529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:25.888529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:25.888529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:25.888529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.888529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:25.888873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.888873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:25.888873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:25.888873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:25.888873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.888873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:25.888873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:25.888873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:25.888873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:25.888873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.888873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:25.889069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:25.889069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:25.889069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.889069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:25.889069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:25.889069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:25.889069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:25.889069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:25.889069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:25.889069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:25.889069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:25.889069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:25.889069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:25.889069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:25.889403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:25.889403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:25.889403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.889403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:25.889584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.889584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:25.889584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:25.889584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:25.889584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:25.889584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:25.889584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.889584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:25.889584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:25.889584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.889584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:25.889584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:25.889584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.889584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:25.889584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:25.889584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.889584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:25.889917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:25.889917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:25.889917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:25.889917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.889917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:25.889917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:25.889917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:25.889917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.889917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:25.889917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:25.889917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:25.889917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:25.889917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:25.889917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:25.889917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:25.889917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:25.889917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:25.889917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:25.889917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:25.890202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:25.890202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:25.890202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.890202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:25.906156 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:25.906448 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:25.906527 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:25.906714 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:25.907265 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:25.907582 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:25.907841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.908431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:25.908431 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:25.908431 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:25.908431 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:25.908431 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:25.908431 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:25.908431 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.908431 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:25.909131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.909131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:25.909131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:25.909131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:25.909131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.909131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:25.909131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:25.909131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.909131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:25.909131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:25.909131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:25.909131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.909539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:25.909539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:25.909539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:25.909539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:25.909539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:25.909539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:25.909539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:25.909539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:25.909539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:25.909539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:25.909539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:25.909539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:25.909539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:25.909958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.909958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:25.909958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.909958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:25.909958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:25.909958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:25.909958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:25.909958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:25.909958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.909958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:25.909958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:25.909958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.909958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:25.909958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:25.909958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.910519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:25.910519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:25.910519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:25.910519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:25.910519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:25.910519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.910519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:25.910519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:25.910519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:25.910519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:25.910519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:25.910519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:25.910519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:25.910519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:25.910519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:25.910519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:25.910519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:25.910519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:25.910519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:25.911039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:25.911039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:25.911039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:25.911039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:25.911039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:25.911039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:28.795436 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:28.796683 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:28.796683 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:28.796838 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:28.824709 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:28.824939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:28.825501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:28.825501 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:28.825501 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:28.825501 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:28.825501 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:28.825501 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:28.825501 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:28.825501 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:28.826215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:28.826215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:28.826215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:28.826215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:28.826215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:28.826215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:28.826215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:28.826215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:28.826215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:28.826215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:28.826215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:28.826215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:28.826514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:28.826514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:28.826514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:28.826514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:28.826514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:28.826514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:28.826514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:28.826514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:28.826514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:28.826514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:28.826514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:28.826514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:28.826514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:28.826775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:28.826775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:28.827178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:28.827178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:28.827178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:28.827178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:28.827178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:28.827178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:28.827178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:28.827178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:28.827178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:28.827178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:28.827178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:28.827178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:28.827178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:28.827178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:28.827178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:28.827178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:28.827613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:28.827613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:28.827613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:28.827613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:28.827613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:28.827613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:28.827613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:28.827613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:28.827613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:28.827613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:28.827613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:28.827613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:28.827613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:28.827613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:28.827613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:28.827613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:28.827613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:28.827613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:28.827613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:28.827952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:28.827952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:28.827952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:30.873742 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:30.874140 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:30.874272 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:30.874910 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:30.875057 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-847f8846-d08f-4c7a-a597-500ef6e7ff78, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:30.875165 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:30.875275 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:30.875386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.875806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:30.875806 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:30.875806 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:30.875806 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:30.875806 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:30.875806 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:30.875806 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.875806 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:30.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:30.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:30.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:30.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:30.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:30.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:30.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:30.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:30.876414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.876671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:30.876671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:30.876671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:30.876671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:30.876671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:30.876671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:30.876671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:30.876671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:30.876671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:30.876671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:30.876671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:30.876671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:30.876671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:30.876907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.876907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:30.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:30.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:30.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:30.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:30.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:30.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:30.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:30.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:30.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:30.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:30.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:30.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:30.878184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:30.878503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:30.878503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.878503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:30.880332 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:30.880572 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:30.880683 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:30.880805 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:42:30.881103 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:30.881302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.881733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:30.881733 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:30.881733 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:30.881733 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:30.881733 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:30.881733 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:30.881733 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.881733 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:30.882382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.882382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:30.882382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:30.882382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:30.882382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.882382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:30.882382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:30.882382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:30.882382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.882382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:30.882382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:30.882382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:30.882382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:30.882382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:30.882382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:30.882938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:30.882938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:30.882938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:30.882938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:30.882938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:30.882938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:30.882938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:30.882938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.882938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:30.884625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.884625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:30.884625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:30.884625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:30.884625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.884625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:30.884625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:30.884625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:42:30.884625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:30.884625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:30.884625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.884625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:30.884951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:30.884951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:30.884951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.884951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:30.884951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:30.884951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:30.884951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:30.884951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:30.884951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:30.884951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:30.884951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:30.884951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:30.884951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:30.884951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:30.885216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:30.885216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:30.885216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.885216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:30.885216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.885216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:30.885216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:30.885216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:30.885216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:30.885216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:30.885216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.885216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:30.885216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:30.885216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.885512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:30.885512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:30.885512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.885512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:30.885512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:30.885512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.885512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:30.885512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:30.885512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:42:30.885512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:30.885512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:30.885512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.885512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:30.885512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:30.885512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:30.885512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.885512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:30.885512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:30.885512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:30.885887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:30.885887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:30.885887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:30.885887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:30.885887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:30.885887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:30.885887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:30.885887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:30.885887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:30.885887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:30.885887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.885887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:30.886394 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:30.886925 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:30.886925 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:30.886925 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:30.887046 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:30.887243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.887683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:30.887683 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:30.887683 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:30.887683 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:30.887683 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:30.887683 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:30.887683 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.887683 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:30.888223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.888223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:30.888223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:30.888223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:30.888223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.888223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:30.888223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:30.888223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:30.888223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.888223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:30.888223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:30.888223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:30.888223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:30.888223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:30.888223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:30.888576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:30.888576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:30.888576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:30.888576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:30.888576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:30.888576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:30.888576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:30.888576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.888576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:30.889206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.889206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:30.889206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:30.889206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:30.889206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.889206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:30.889206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:30.889206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:30.889206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:30.889206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.889206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:30.889470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:30.889470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:30.889470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.889470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:30.889470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:30.889470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:30.889470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:30.889470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:30.889470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:30.889470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:30.889470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:30.889470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:30.889470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:30.889470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:30.889729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:30.889729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:30.889729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.889729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:30.890158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.890158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:30.890158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:30.890158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:30.890158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:30.890158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:30.890158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.890158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:30.890158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:30.890158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.890158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:30.890158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:30.890158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.890158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:30.890158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:30.890158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.890158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:30.890549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:30.890549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:30.890549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:30.890549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.890549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:30.890549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:30.890549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:30.890549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.890549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:30.890549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:30.890549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:30.890549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:30.890549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:30.890549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:30.890549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:30.890549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:30.890549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:30.890549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:30.890549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:30.890915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:30.890915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:30.890915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.890915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:30.906534 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:30.906791 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:30.906890 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:30.907074 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:30.907338 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:30.907551 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:30.907723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.908177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:30.908177 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:30.908177 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:30.908177 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:30.908177 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:30.908177 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:30.908177 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.908177 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:30.908847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.908847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:30.908847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:30.908847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:30.908847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.908847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:30.908847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:30.908847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.908847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:30.908847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:30.908847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:30.908847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.909198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:30.909198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:30.909198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:30.909198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:30.909198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:30.909198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:30.909198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:30.909198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:30.909198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:30.909198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:30.909198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:30.909198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:30.909198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:30.909530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.909530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:30.909830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.909830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:30.909830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:30.909830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:30.909830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:30.909830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:30.909830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.909830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:30.909830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:30.909830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.909830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:30.909830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:30.909830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.909830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:30.909830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:30.909830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:30.910120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:30.910120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:30.910120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.910120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:30.910120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:30.910120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:30.910120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:30.910120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:30.910120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:30.910120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:30.910120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:30.910120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:30.910120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:30.910120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:30.910120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:30.910120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:30.910120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:30.910120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:30.910120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:30.910444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:30.910444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:30.910444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:33.796288 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:33.798319 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:33.798319 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:33.798319 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:33.829432 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:33.829744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:33.831486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:33.831486 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:33.831486 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:33.831486 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:33.831486 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:33.831486 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:33.831486 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:33.831486 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:33.831486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:33.831486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:33.831486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:33.831486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:33.831486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:33.831486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:33.831486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:33.831866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:33.831866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:33.831866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:33.831866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:33.831866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:33.831866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:33.831866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:33.831866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:33.831866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:33.831866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:33.831866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:33.831866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:33.831866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:33.831866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:33.832108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:33.832108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:33.832108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:33.832108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:33.832108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:33.832108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:33.832215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:33.832215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:33.832215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:33.832215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:33.832215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:33.832215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:33.832215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:33.832215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:33.832215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:33.832215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:33.832215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:33.832215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:33.832215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:33.832215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:33.832215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:33.832215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:33.832519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:33.832519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:33.832519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:33.832519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:33.832519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:33.832519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:33.832519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:33.832519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:33.832519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:33.832519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:33.832519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:33.832519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:33.832519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:33.832519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:33.832519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:33.832519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:33.832519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:33.832519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:33.832519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:33.832828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:33.832828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:33.832828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:35.874167 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:35.874661 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:35.874661 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:35.875328 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:35.875541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:35.875705 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:35.875867 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:35.876008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:35.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:35.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:35.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:35.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:35.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:35.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.876513 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:35.877187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.877187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:35.877187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:35.877187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:35.877187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.877187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:35.877187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:35.877187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.877187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:35.877187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:35.877187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:35.877187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.877472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:35.877472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:35.877472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:35.877472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:35.877472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:35.877472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:35.877472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:35.877472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:35.877472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:35.877472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:35.877472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:35.877472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:35.877472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:35.877722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.877722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:35.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:35.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:35.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:35.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:35.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:35.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:35.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:35.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:35.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:35.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:35.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:35.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.878268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:35.878268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:35.878268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.878268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:35.878268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:35.878268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:35.878268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.878268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:35.878268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:35.878268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:35.878268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:35.878268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:35.878268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:35.878268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:35.878268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:35.878268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:35.878268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:35.878268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:35.878268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:35.878631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:35.878631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.878631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:35.880614 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:35.880812 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:35.880898 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:35.880987 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:42:35.881262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:35.881417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.881785 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:35.882299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.882299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:35.882299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:35.882299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:35.882299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.882299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:35.882299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:35.882299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:35.882299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.882299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:35.882299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:35.882299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:35.882299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:35.882299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:35.882299 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:35.882580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:35.882580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:35.882580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:35.882580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:35.882580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:35.882580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:35.882580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:35.882580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.882580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:35.883203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.883203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:35.883203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:35.883203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:35.883203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.883203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:35.883203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:35.883203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:42:35.883203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:35.883203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:35.883203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.883203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:35.883469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:35.883469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:35.883469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.883469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:35.883469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:35.883469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:35.883469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:35.883469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:35.883469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:35.883469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:35.883469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:35.883469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:35.883469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:35.883469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:35.883690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:35.883690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:35.883690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.883690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:35.883927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.883927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:35.883927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:35.883927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:35.883927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:35.883927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:35.883927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.883927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:35.883927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:35.883927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.883927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:35.883927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:35.883927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.883927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:35.883927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:35.883927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.884190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:35.884190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:35.884190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:42:35.884190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:35.884190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:35.884190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.884190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:35.884190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:35.884190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:35.884190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.884190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:35.884190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:35.884190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:35.884190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:35.884190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:35.884190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:35.884190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:35.884190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:35.884511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:35.884511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:35.884511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:35.884511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:35.884511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:35.884511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.884511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:35.886673 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:35.886870 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:35.886948 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:35.887057 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:35.887336 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:35.887517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.887836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:35.887836 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:35.887836 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:35.887836 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:35.887836 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:35.887836 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:35.887836 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.887836 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:35.888365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.888365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:35.888365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:35.888365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:35.888365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.888365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:35.888365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:35.888365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:35.888365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.888365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:35.888365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:35.888365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:35.888365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:35.888365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:35.888365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:35.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:35.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:35.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:35.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:35.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:35.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:35.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:35.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.888659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:35.888991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.888991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:35.888991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:35.888991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:35.888991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.888991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:35.888991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:35.888991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:35.888991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:35.888991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.888991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:35.889198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:35.889198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:35.889198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.889198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:35.889198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:35.889198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:35.889198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:35.889198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:35.889198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:35.889198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:35.889198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:35.889198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:35.889198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:35.889198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:35.889428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:35.889428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:35.889428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.889428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:35.889669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.889669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:35.889669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:35.889669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:35.889669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:35.889669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:35.889669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.889669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:35.889669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:35.889669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.889669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:35.889669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:35.889669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.889669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:35.889669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:35.889669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.889669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:35.889934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:35.889934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:35.889934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:35.889934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.889934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:35.889934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:35.889934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:35.889934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.889934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:35.889934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:35.889934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:35.889934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:35.889934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:35.889934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:35.889934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:35.889934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:35.889934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:35.889934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:35.889934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:35.890222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:35.890222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:35.890222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.890222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:35.906947 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:35.907162 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:35.907246 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:35.907392 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:35.907753 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:35.908028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:35.908219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.908822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:35.908822 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:35.908822 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:35.908822 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:35.908822 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:35.908822 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:35.908822 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.908822 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:35.909399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.909399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:35.909399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:35.909399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:35.909399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.909399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:35.909399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:35.909399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.909399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:35.909399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:35.909399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:35.909399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.909707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:35.909707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:35.909707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:35.909707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:35.909707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:35.909707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:35.909707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:35.909707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:35.909707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:35.909707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:35.909707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:35.909707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:35.909707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:35.910061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.910061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:35.910406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.910406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:35.910406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:35.910406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:35.910406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:35.910406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:35.910406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.910406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:35.910406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:35.910406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.910406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:35.910406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:35.910406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.910406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:35.910406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:35.910406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:35.910894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:35.910894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:35.910894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.910894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:35.910894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:35.910894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:35.910894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:35.910894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:35.910894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:35.910894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:35.910894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:35.910894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:35.910894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:35.910894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:35.910894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:35.910894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:35.910894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:35.910894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:35.910894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:35.911699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:35.911699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:35.911699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:38.797668 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:38.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:38.798134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:38.798831 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:38.822981 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:38.823134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:38.823613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:38.823613 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:38.823613 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:38.823613 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:38.823613 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:38.823613 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:38.823613 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:38.823613 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:38.824191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:38.824191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:38.824191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:38.824191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:38.824191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:38.824191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:38.824191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:38.824191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:38.824191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:38.824191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:38.824191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:38.824191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:38.824477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:38.824477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:38.824477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:38.824477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:38.824477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:38.824477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:38.824477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:38.824477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:38.824477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:38.824477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:38.824477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:38.824477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:38.824477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:38.824756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:38.824756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:38.824954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:38.824954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:38.824954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:38.824954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:38.824954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:38.824954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:38.824954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:38.824954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:38.824954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:38.824954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:38.824954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:38.824954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:38.824954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:38.824954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:38.824954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:38.824954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:38.825272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:38.825272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:38.825272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:38.825272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:38.825272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:38.825272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:38.825272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:38.825272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:38.825272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:38.825272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:38.825272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:38.825272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:38.825272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:38.825272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:38.825272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:38.825272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:38.825272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:38.825272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:38.825272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:38.825634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:38.825634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:38.825634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:40.874567 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:40.875076 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:40.875076 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:40.875718 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:40.875888 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:40.876025 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:40.876151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:40.876296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.876782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:40.876782 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:40.876782 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:40.876782 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:40.876782 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:40.876782 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:40.876782 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.876782 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:40.877555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.877555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:40.877555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:40.877555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:40.877555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.877555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:40.877555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:40.877555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.877555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:40.877555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:40.877555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:40.877555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:40.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:40.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:40.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:40.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:40.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:40.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:40.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:40.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:40.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:40.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:40.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:40.877965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:40.878355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.878355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:40.878649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.878649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:40.878649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:40.878649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:40.878649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:40.878649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:40.878649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.878649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:40.878649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:40.878649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.878649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:40.878649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:40.878649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.878649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:40.878649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:40.878649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.878960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:40.878960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:40.878960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.878960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:40.878960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:40.878960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:40.878960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.878960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:40.878960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:40.878960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:40.878960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:40.878960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:40.878960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:40.878960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:40.878960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:40.878960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:40.878960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:40.878960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:40.878960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:40.879345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:40.879345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.879345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:40.880824 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:40.881022 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:40.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:40.881271 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:42:40.881518 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:40.881695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.882034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:40.882034 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:40.882034 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:40.882034 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:40.882034 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:40.882034 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:40.882034 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.882034 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:40.882721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.882721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:40.882721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:40.882721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:40.882721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.882721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:40.882721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:40.882721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:40.882721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.882721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:40.882721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:40.882721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:40.882721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:40.882721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:40.882721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:40.883185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:40.883185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:40.883185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:40.883185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:40.883185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:40.883185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:40.883185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:40.883185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.883185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:40.883629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.883629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:40.883629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:40.883629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:40.883629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.883629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:40.883629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:40.883629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:42:40.883629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:40.883629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:40.883629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.883629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:40.883991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:40.883991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:40.883991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.883991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:40.883991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:40.883991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:40.883991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:40.883991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:40.883991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:40.883991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:40.883991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:40.883991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:40.883991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:40.883991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:40.884395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:40.884395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:40.884395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.884395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:40.884719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.884719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:40.884719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:40.884719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:40.884719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:40.884719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:40.884719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.884719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:40.884719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:40.884719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.884719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:40.884719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:40.884719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.884719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:40.884719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:40.884719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.885043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:40.885043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:40.885043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:42:40.885043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:40.885043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:40.885043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.885043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:40.885043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:40.885043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:40.885043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.885043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:40.885043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:40.885043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:40.885043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:40.885043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:40.885043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:40.885043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:40.885043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:40.885363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:40.885363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:40.885363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:40.885363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:40.885363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:40.885363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.885363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:40.886910 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:40.887066 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:40.887190 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:40.887338 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:40.887633 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:40.887835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.888171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:40.888171 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:40.888171 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:40.888171 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:40.888171 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:40.888171 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:40.888171 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.888171 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:40.888791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.888791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:40.888791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:40.888791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:40.888791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.888791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:40.888791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:40.888791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:40.888791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.888791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:40.888791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:40.888791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:40.888791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:40.888791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:40.888791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:40.889210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:40.889210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:40.889210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:40.889210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:40.889210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:40.889210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:40.889210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:40.889210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.889210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:40.889697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.889697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:40.889697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:40.889697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:40.889697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.889697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:40.889697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:40.889697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:40.889697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:40.889697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.889697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:40.890028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:40.890028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:40.890028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.890028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:40.890028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:40.890028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:40.890028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:40.890028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:40.890028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:40.890028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:40.890028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:40.890028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:40.890028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:40.890028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:40.890415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:40.890415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:40.890415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.890415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:40.890814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.890814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:40.890814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:40.890814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:40.890814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:40.890814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:40.890814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.890814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:40.890814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:40.890814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.890814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:40.890814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:40.890814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.890814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:40.890814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:40.890814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.890814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:40.891184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:40.891184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:40.891184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:40.891184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.891184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:40.891184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:40.891184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:40.891184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.891184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:40.891184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:40.891184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:40.891184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:40.891184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:40.891184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:40.891184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:40.891184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:40.891184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:40.891184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:40.891184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:40.891498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:40.891498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:40.891498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.891498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:40.907238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:40.907483 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:40.907619 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:40.907790 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:40.908044 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:40.908268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:40.908609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.908900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:40.908900 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:40.908900 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:40.908900 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:40.908900 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:40.908900 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:40.908900 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.908900 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:40.909605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.909605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:40.909605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:40.909605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:40.909605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.909605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:40.909605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:40.909605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.909605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:40.909605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:40.909605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:40.909605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.910043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:40.910043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:40.910043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:40.910043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:40.910043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:40.910043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:40.910043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:40.910043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:40.910043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:40.910043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:40.910043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:40.910043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:40.910043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:40.910431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.910431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:40.910431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.910431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:40.910431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:40.910431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:40.910431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:40.910431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:40.910431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.910431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:40.910431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:40.910431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.910431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:40.910431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:40.910431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.910907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:40.910907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:40.910907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:40.910907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:40.910907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:40.910907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.910907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:40.910907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:40.910907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:40.910907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:40.910907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:40.910907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:40.910907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:40.910907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:40.910907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:40.910907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:40.910907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:40.910907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:40.910907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:40.911475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:40.911475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:40.911475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:40.911475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:40.911475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:40.911475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:43.798155 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:43.798616 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:43.798616 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:43.800111 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:43.826292 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:43.826572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:43.827180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:43.827180 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:43.827180 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:43.827180 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:43.827180 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:43.827180 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:43.827180 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:43.827180 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:43.828210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:43.828210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:43.828210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:43.828210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:43.828210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:43.828210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:43.828210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:43.828210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:43.828210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:43.828210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:43.828210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:43.828210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:43.828484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:43.828484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:43.828484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:43.828484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:43.828484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:43.828484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:43.828484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:43.828484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:43.828484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:43.828484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:43.828484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:43.828484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:43.828484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:43.828718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:43.828718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:43.829337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:43.829337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:43.829337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:43.829337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:43.829337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:43.829337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:43.829337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:43.829337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:43.829337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:43.829337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:43.829337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:43.829337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:43.829337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:43.829337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:43.829337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:43.829337 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:43.829665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:43.829665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:43.829665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:43.829665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:43.829665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:43.829665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:43.829665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:43.829665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:43.829665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:43.829665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:43.829665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:43.829665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:43.829665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:43.829665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:43.829665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:43.829665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:43.829665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:43.829665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:43.829665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:43.830142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:43.830142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:43.830142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:45.875070 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:45.875585 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:45.875631 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:45.876544 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:45.876991 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:45.877126 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:45.877298 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:45.877474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.879335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:45.879335 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:45.879335 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:45.879335 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:45.879335 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:45.879335 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:45.879335 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.879335 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:45.879335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.879335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:45.879335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:45.879335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:45.879335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.879335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:45.879335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:45.879985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.879985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:45.879985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:45.879985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:45.879985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.879985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:45.879985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:45.879985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:45.879985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:45.879985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:45.879985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:45.879985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:45.879985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:45.879985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:45.880400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:45.880400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:45.880400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:45.880400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:45.880400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.880400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:45.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:45.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:45.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:45.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:45.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:45.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:45.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:45.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:45.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:45.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:45.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:45.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.881049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:45.881049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:45.881049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.881049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:45.881049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:45.881049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:45.881049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.881049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:45.881049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:45.881049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:45.881049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:45.881049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:45.881049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:45.881049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:45.881049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:45.881049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:45.881049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:45.881049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:45.881049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:45.881581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:45.881581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.881581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:45.881581 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:45.881581 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:45.881581 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:45.881581 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:42:45.881936 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:45.882088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.882598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:45.882598 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:45.882598 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:45.882598 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:45.882598 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:45.882598 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:45.882598 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.882598 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:45.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:45.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:45.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:45.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:45.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:45.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:45.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:45.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:45.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:45.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:45.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:45.883121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:45.883613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:45.883613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:45.883613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:45.883613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:45.883613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:45.883613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:45.883613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:45.883613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.883613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:45.883847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.883847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:45.883847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:45.883847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:45.883847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.883847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:45.883847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:45.883847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:42:45.883847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:45.883847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:45.883847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.883847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:45.884177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:45.884177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:45.884177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.884177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:45.884177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:45.884177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:45.884177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:45.884177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:45.884177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:45.884177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:45.884177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:45.884177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:45.884177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:45.884177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:45.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:45.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:45.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:45.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:45.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:45.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:45.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:45.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:45.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:45.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:45.884516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.884857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:45.884857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:45.884857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.884857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:45.884857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:45.884857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.884857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:45.884857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:45.884857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:42:45.884857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:45.884857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:45.884857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.884857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:45.884857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:45.884857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:45.884857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.884857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:45.884857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:45.884857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:45.885352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:45.885352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:45.885352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:45.885352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:45.885352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:45.885352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:45.885352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:45.885352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:45.885352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:45.885352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:45.885352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.885352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:45.887170 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:45.887404 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:45.887532 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:45.887641 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:45.887972 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:45.888271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.888870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:45.888870 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:45.888870 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:45.888870 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:45.888870 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:45.888870 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:45.888870 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.888870 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:45.889883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.889883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:45.889883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:45.889883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:45.889883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.889883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:45.889883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:45.889883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:45.889883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.889883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:45.889883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:45.889883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:45.889883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:45.889883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:45.889883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:45.890293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:45.890293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:45.890293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:45.890293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:45.890293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:45.890293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:45.890293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:45.890293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.890293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:45.890848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.890848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:45.890848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:45.890848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:45.890848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.890848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:45.890848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:45.890848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:45.890848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:45.890848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.890848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:45.891098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:45.891098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:45.891098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.891098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:45.891098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:45.891098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:45.891098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:45.891098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:45.891098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:45.891098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:45.891098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:45.891098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:45.891098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:45.891098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:45.891353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:45.891353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:45.891353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.891353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:45.892090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.892090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:45.892090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:45.892090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:45.892090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:45.892090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:45.892090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.892090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:45.892090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:45.892090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.892090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:45.892090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:45.892090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.892090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:45.892090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:45.892090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.892090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:45.892583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:45.892583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:45.892583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:45.892583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.892583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:45.892583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:45.892583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:45.892583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.892583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:45.892583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:45.892583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:45.892583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:45.892583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:45.892583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:45.892583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:45.892583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:45.892583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:45.892583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:45.892583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:45.892888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:45.892888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:45.892888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.892888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:45.907577 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:45.907763 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:45.907846 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:45.908002 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:45.908397 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:45.908643 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:45.908881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.909379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:45.909379 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:45.909379 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:45.909379 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:45.909379 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:45.909379 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:45.909379 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.909379 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:45.910065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.910065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:45.910065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:45.910065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:45.910065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.910065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:45.910065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:45.910065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.910065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:45.910065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:45.910065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:45.910065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.910401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:45.910401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:45.910401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:45.910401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:45.910401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:45.910401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:45.910401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:45.910401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:45.910401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:45.910401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:45.910401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:45.910401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:45.910401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:45.911077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.911077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:45.911077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.911077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:45.911077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:45.911077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:45.911077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:45.911077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:45.911077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.911077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:45.911077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:45.911077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.911077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:45.911077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:45.911077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.911361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:45.911361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:45.911361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:45.911361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:45.911361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:45.911361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.911361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:45.911361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:45.911361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:45.911361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:45.911361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:45.911361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:45.911361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:45.911361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:45.911361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:45.911361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:45.911361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:45.911361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:45.911361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:45.911699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:45.911699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:45.911699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:45.911699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:45.911699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:45.911699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:48.798526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:48.799075 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:48.799075 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:48.799515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:48.825860 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:48.826069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:48.826655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:48.826655 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:48.826655 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:48.826655 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:48.826655 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:48.826655 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:48.826655 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:48.826655 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:48.827379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:48.827379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:48.827379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:48.827379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:48.827379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:48.827379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:48.827379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:48.827379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:48.827379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:48.827379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:48.827379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:48.827379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:48.827740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:48.827740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:48.827740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:48.827740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:48.827740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:48.827740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:48.827740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:48.827740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:48.827740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:48.827740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:48.827740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:48.827740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:48.827740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:48.828091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:48.828091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:48.828160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:48.828160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:48.828160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:48.828160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:48.828160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:48.828160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:48.828160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:48.828160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:48.828160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:48.828160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:48.828160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:48.828160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:48.828160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:48.828160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:48.828160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:48.828160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:48.828614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:48.828614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:48.828614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:48.828614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:48.828614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:48.828614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:48.828614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:48.828614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:48.828614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:48.828614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:48.828614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:48.828614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:48.828614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:48.828614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:48.828614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:48.828614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:48.828614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:48.828614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:48.828614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:48.829174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:48.829174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:48.829174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:50.875510 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:50.875992 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:50.875992 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:50.876517 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:50.876685 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:50.876799 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:50.876906 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:50.877016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.877432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:50.877432 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:50.877432 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:50.877432 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:50.877432 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:50.877432 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:50.877432 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.877432 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:50.878077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.878077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:50.878077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:50.878077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:50.878077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.878077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:50.878077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:50.878077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.878077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:50.878077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:50.878077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:50.878077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.878323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:50.878323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:50.878323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:50.878323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:50.878323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:50.878323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:50.878323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:50.878323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:50.878323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:50.878323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:50.878323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:50.878323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:50.878323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:50.878548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.878548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:50.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:50.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:50.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:50.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:50.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:50.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:50.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:50.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:50.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:50.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:50.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:50.878823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.879109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:50.879109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:50.879109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.879109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:50.879109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:50.879109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:50.879109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.879109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:50.879109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:50.879109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:50.879109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:50.879109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:50.879109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:50.879109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:50.879109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:50.879109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:50.879109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:50.879109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:50.879109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:50.879431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:50.879431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.879431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:50.881396 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:50.881600 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:50.881698 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:50.881810 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:42:50.882145 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:50.882422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.882837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:50.882837 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:50.882837 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:50.882837 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:50.882837 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:50.882837 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:50.882837 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.882837 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:50.883612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.883612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:50.883612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:50.883612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:50.883612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.883612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:50.883612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:50.883612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:50.883612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.883612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:50.883612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:50.883612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:50.883612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:50.883612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:50.883612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:50.883894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:50.883894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:50.883894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:50.883894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:50.883894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:50.883894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:50.883894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:50.883894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.883894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:50.884264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.884264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:50.884264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:50.884264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:50.884264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.884264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:50.884264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:50.884264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:42:50.884264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:50.884264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:50.884264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.884264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:50.884509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:50.884509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:50.884509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.884509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:50.884509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:50.884509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:50.884509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:50.884509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:50.884509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:50.884509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:50.884509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:50.884509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:50.884509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:50.884509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:50.884749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:50.884749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:50.884749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.884749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:50.885166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.885166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:50.885166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:50.885166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:50.885166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:50.885166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:50.885166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.885166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:50.885166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:50.885166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.885166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:50.885166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:50.885166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.885166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:50.885166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:50.885166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:50.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:50.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:42:50.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:50.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:50.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:50.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:50.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:50.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:50.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:50.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:50.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:50.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:50.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:50.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:50.885467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:50.885840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:50.885840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:50.885840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:50.885840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:50.885840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:50.885840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.885840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:50.887410 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:50.887542 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:50.887633 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:50.887726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:50.887973 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:50.888173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.888548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:50.888548 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:50.888548 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:50.888548 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:50.888548 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:50.888548 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:50.888548 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.888548 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:50.889082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.889082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:50.889082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:50.889082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:50.889082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.889082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:50.889082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:50.889082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:50.889082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.889082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:50.889082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:50.889082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:50.889082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:50.889082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:50.889082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:50.889343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:50.889343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:50.889343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:50.889343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:50.889343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:50.889343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:50.889343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:50.889343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.889343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:50.889675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.889675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:50.889675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:50.889675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:50.889675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.889675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:50.889675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:50.889675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:50.889675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:50.889675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.889675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:50.889870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:50.889870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:50.889870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.889870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:50.889870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:50.889870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:50.889870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:50.889870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:50.889870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:50.889870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:50.889870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:50.889870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:50.889870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:50.889870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:50.890137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:50.890137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:50.890137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.890137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:50.890372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.890372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:50.890372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:50.890372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:50.890372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:50.890372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:50.890372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.890372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:50.890372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:50.890372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.890372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:50.890372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:50.890372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.890372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:50.890372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:50.890372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.890372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:50.890830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:50.890830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:50.890830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:50.890830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.890830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:50.890830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:50.890830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:50.890830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.890830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:50.890830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:50.890830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:50.890830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:50.890830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:50.890830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:50.890830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:50.890830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:50.890830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:50.890830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:50.890830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:50.891372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:50.891372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:50.891372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.891372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:50.907949 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:50.908147 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:50.908233 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:50.908381 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:50.908690 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:50.908891 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:50.909065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.909479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:50.909479 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:50.909479 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:50.909479 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:50.909479 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:50.909479 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:50.909479 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.909479 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:50.910040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.910040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:50.910040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:50.910040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:50.910040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.910040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:50.910040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:50.910040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.910040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:50.910040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:50.910040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:50.910040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.910298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:50.910298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:50.910298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:50.910298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:50.910298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:50.910298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:50.910298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:50.910298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:50.910298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:50.910298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:50.910298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:50.910298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:50.910298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:50.910504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.910504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:50.910759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.910759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:50.910759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:50.910759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:50.910759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:50.910759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:50.910759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.910759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:50.910759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:50.910759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.910759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:50.910759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:50.910759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.910759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:50.910759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:50.910759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:50.911165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:50.911165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:50.911165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.911165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:50.911165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:50.911165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:50.911165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:50.911165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:50.911165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:50.911165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:50.911165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:50.911165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:50.911165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:50.911165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:50.911165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:50.911165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:50.911165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:50.911165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:50.911165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:50.911728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:50.911728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:50.911728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:53.798869 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:53.799207 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:53.799207 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:53.799877 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:53.811042 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=240 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:42:53.841926 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:53.842119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:53.842587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:53.842587 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:53.842587 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:53.842587 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:53.842587 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:53.842587 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:53.842587 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:53.842587 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:53.843300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:53.843300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:53.843300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:53.843300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:53.843300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:53.843300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:53.843300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:53.843300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:53.843300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:53.843300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:53.843300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:53.843300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:53.843763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:53.843763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:53.843763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:53.843763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:53.843763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:53.843763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:53.843763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:53.843763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:53.843763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:53.843763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:53.843763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:53.843763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:53.843763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:53.844097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:53.844097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:53.844368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:53.844368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:53.844368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:53.844368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:53.844368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:53.844368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:53.844368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:53.844368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:53.844368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:53.844368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:53.844368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:53.844368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:53.844368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:53.844368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:53.844368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:53.844368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:53.844697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:53.844697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:53.844697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:53.844697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:53.844697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:53.844697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:53.844697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:53.844697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:53.844697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:53.844697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:53.844697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:53.844697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:53.844697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:53.844697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:53.844697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:53.844697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:53.844697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:53.844697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:53.844697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:53.844986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:53.844986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:53.844986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:55.875893 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:55.880660 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:55.880660 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:55.880660 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:55.880660 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:55.880660 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:55.882298 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:55.882298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.882298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:55.882298 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:55.882298 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:55.882298 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:55.882298 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:55.882298 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:55.882298 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.882298 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:55.882298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.882298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:55.882298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:55.882298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:55.884456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.884456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:55.884456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:55.884456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.884456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:55.884456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:55.884456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:55.884456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.884456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:55.884456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:55.884456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:55.884456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:55.884456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:55.884456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:55.885604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:55.885604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:55.885604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:55.885604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:55.885604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:55.885604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:55.885604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:55.885604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.885604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:55.885604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.885604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:55.885604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:55.885931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.887159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:55.887159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:55.887159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:55.887159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:55.887159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:55.887159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:55.887159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:55.887159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:55.887159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:55.887159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:55.887159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:55.887159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:55.887159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:55.887159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.887159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:55.887159 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:55.887653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:55.887653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:55.887653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:42:55.887653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:55.887653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.887653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:55.887653 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:55.887653 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:55.887653 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:55.887868 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:55.887868 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:55.887868 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.887868 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:55.887868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.887868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:55.887868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:55.887868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:55.887868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.887868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:55.887868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:55.887868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:55.887868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.887868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:55.887868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:55.887868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:55.888973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:55.888973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:55.888973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:55.888973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:55.888973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:55.888973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:55.888973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:55.888973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:55.888973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:55.888973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:55.888973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.888973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:55.888973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.888973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:55.890052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:55.890052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:55.890052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.890052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:55.890052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:55.890052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:42:55.890052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:55.890052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:55.890052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.890052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:55.890052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:55.890052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:55.890052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.890379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:55.890379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:55.890379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:55.890379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:55.890379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:55.890379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:55.890379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:55.890379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:55.890379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:55.890379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:55.890379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:55.890379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:55.890379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:55.890647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.890647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:55.890647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.890647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:55.890647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:55.890647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:55.890647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:55.890647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:55.890647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.890647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:55.890647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:55.890647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.890647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:55.890647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:55.890647 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.890895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:55.890895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:55.890895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.890895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:42:55.890895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:42:55.890895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:42:55.890895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:55.890895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:55.890895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.890895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:55.890895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:55.890895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:55.890895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.890895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:55.890895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:55.890895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:55.890895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:55.890895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:55.890895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:55.891286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:55.891286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:55.891286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:55.891286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:55.891286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:55.891286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:55.891286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:55.891286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.891286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:55.891286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:55.891286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:55.891286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:55.891528 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:55.891528 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:55.891528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.891528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:55.891528 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:55.891528 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:55.891528 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:55.891528 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:55.891528 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:55.891528 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.891528 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:55.891528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.891837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:42:55.891837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:55.891837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:55.891837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.891837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:55.891837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:55.891837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:42:55.891837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.891837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:42:55.891837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:55.891837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:42:55.891837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:55.891837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:42:55.891837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:42:55.891837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:55.891837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:42:55.891837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:55.892117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:42:55.892117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:42:55.892117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:55.892117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:42:55.892117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.892117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:42:55.892117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.892117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:55.892117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:55.892117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:55.892117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.892117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:55.892117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:55.892784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:55.892784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:55.892784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.892784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:55.892784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:55.892784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:55.892784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.892784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:55.892784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:55.892784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:55.892784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:55.892784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:55.892784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:55.892999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:55.892999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:55.892999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:55.892999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:55.892999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:55.892999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:55.892999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:55.892999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.892999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:55.892999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.892999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:55.892999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:55.893363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:55.893363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:55.893363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:55.893363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.893363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:55.893363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:55.893363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.893363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:55.893363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:55.893363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.893363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:55.893363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:55.893363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.893363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:42:55.893363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:42:55.893363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:42:55.893363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:42:55.893363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.893363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:55.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:55.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:55.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:55.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:55.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:55.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:55.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:55.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:55.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:55.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:55.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:55.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:55.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:55.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:55.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:55.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:55.908272 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:55.908513 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:55.908624 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:55.908786 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:55.909008 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:55.909211 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:55.909445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.909777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:55.909777 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:55.909777 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:55.909777 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:55.909777 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:55.909777 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:55.909777 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.909777 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:55.910429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.910429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:55.910429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:55.910429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:55.910429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.910429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:55.910429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:55.910429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.910429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:55.910429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:55.910429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:55.910429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.910849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:55.910849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:55.910849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:55.910849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:55.910849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:55.910849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:55.910849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:55.910849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:55.910849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:55.910849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:55.910849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:55.910849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:55.910849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:55.911081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.911081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:55.911401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.911401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:55.911401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:55.911401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:55.911401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:55.911401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:55.911401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.911401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:55.911401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:55.911401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.911401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:55.911401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:55.911401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.911401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:55.911401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:55.911401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:55.911727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:42:55.911727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:55.911727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.911727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:55.911727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:55.911727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:55.911727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:55.911727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:55.911727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:55.911727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:55.911727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:55.911727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:55.911727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:55.911727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:55.911727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:55.911727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:55.911727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:55.911727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:55.911727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:55.912055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:55.912055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:55.912055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:42:58.799382 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:42:58.799930 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:42:58.799930 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:42:58.800522 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:42:58.827876 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:42:58.828084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:58.828650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:42:58.828650 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:58.828650 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:42:58.828650 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:42:58.828650 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:58.828650 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:42:58.828650 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:58.828650 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:42:58.829758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:58.829758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:42:58.829758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:58.829758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:42:58.829758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:58.829758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:58.829758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:58.829758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:58.829758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:58.829758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:58.829758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:42:58.829758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:58.831317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:42:58.831317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:58.831317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:42:58.831317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:58.831317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:42:58.831317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:42:58.831317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:58.831317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:42:58.831317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:58.831317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:42:58.831317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:42:58.831317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:58.831317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:42:58.832139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:58.832139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:42:58.832139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:58.832139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:42:58.832139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:42:58.832139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:42:58.832139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:42:58.832139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:42:58.832139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:58.832139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:42:58.832139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:42:58.832139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:58.832139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:42:58.832139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:42:58.832139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:42:58.832891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:42:58.832891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:42:58.832891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:42:58.832891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:42:58.832891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:42:58.832891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:58.832891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:58.832891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:42:58.832891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:42:58.832891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:42:58.832891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:42:58.832891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:42:58.832891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:42:58.832891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:42:58.832891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:42:58.832891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:42:58.832891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:42:58.832891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:42:58.832891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:42:58.833413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:42:58.833413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:42:58.833413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:42:58.833413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:42:58.833413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:42:58.833413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:00.876250 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:00.880508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:00.880508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:00.880508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:00.880508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:00.880508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:00.881693 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:00.881693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.881693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:00.881693 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:00.881693 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:00.881693 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:00.881693 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:00.881693 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:00.881693 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.881693 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:00.881693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.881693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:00.881693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:00.881693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:00.883500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.883500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:00.883500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:00.883500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.883500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:00.883500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:00.883500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:00.883500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.883500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:00.883500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:00.883500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:00.883500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:00.883500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:00.883500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:00.884180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:00.884180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:00.884180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:00.884180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:00.884180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:00.884180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:00.884180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:00.884180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.884180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:00.884180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.884180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:00.884180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:00.884832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.886077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:00.886077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:00.886077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:00.886077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:00.886077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:00.886077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:00.886077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:00.886077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:00.886077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:00.886077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:00.886077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:00.886077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:00.886077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:00.886077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.886077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:00.886077 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:00.886530 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:00.886530 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:00.886530 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:43:00.886530 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:00.886530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.886530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:00.886530 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:00.886530 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:00.886530 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:00.886770 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:00.886770 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:00.886770 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.886770 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:00.886770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.886770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:00.886770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:00.886770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:00.886770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.886770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:00.886770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:00.886770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:00.886770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.886770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:00.886770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:00.886770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:00.888011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:00.888011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:00.888011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:00.888011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:00.888011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:00.888011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:00.888011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:00.888011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:00.888011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:00.888011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:00.888011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.888011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:00.888011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.888011 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:00.888922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:00.888922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:00.888922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.888922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:00.888922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:00.888922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:43:00.888922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:00.888922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:00.888922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.888922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:00.888922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:00.888922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:00.888922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.889232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:00.889232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:00.889232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:00.889232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:00.889232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:00.889232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:00.889232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:00.889232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:00.889232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:00.889232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:00.889232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:00.889232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:00.889232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:00.889494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.889494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:00.889494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.889494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:00.889494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:00.889494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:00.889494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:00.889494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:00.889494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.889494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:00.889494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:00.889494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.889494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:00.889494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:00.889494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.889802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:00.889802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:00.889802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.889802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:00.889802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:00.889802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:43:00.889802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:00.889802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:00.889802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.889802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:00.889802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:00.889802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:00.889802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.889802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:00.889802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:00.889802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:00.889802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:00.889802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:00.889802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:00.890262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:00.890262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:00.890262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:00.890262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:00.890262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:00.890262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:00.890262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:00.890262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.890262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:00.890262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:00.890262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:00.890262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:00.890483 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:00.890483 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:00.890483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.890483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:00.890483 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:00.890483 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:00.890483 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:00.890483 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:00.890483 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:00.890483 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.890483 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:00.890483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.890774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:00.890774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:00.890774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:00.890774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.890774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:00.890774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:00.890774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:00.890774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.890774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:00.890774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:00.890774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:00.890774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:00.890774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:00.890774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:00.890774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:00.890774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:00.890774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:00.891068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:00.891068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:00.891068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:00.891068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:00.891068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.891068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:00.891068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.891068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:00.891068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:00.891068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:00.891068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.891068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:00.891068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:00.891741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:00.891741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:00.891741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.891741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:00.891741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:00.891741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:00.891741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.891741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:00.891741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:00.891741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:00.891741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:00.891741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:00.891741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:00.892006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:00.892006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:00.892006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:00.892006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:00.892006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:00.892006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:00.892006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:00.892006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.892006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:00.892006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.892006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:00.892006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:00.892318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:00.892318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:00.892318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:00.892318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.892318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:00.892318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:00.892318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.892318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:00.892318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:00.892318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.892318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:00.892318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:00.892318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.892318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:00.892318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:00.892318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:00.892318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:00.892318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.892318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:00.892974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:00.892974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:00.892974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.892974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:00.892974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:00.892974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:00.892974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:00.892974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:00.892974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:00.892974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:00.892974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:00.892974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:00.892974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:00.892974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:00.892974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:00.892974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:00.892974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.892974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:00.908579 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:00.908816 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:00.908927 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:00.909080 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:00.909407 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:00.909619 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:00.909758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.910161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:00.910161 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:00.910161 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:00.910161 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:00.910161 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:00.910161 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:00.910161 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.910161 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:00.910995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.910995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:00.910995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:00.910995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:00.910995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.910995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:00.910995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:00.910995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.910995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:00.910995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:00.910995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:00.910995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.911664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:00.911664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:00.911664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:00.911664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:00.911664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:00.911664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:00.911664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:00.911664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:00.911664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:00.911664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:00.911664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:00.911664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:00.911664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:00.911879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.911879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:00.912020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.912020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:00.912020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:00.912020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:00.912020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:00.912020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:00.912020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.912020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:00.912020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:00.912020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.912020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:00.912020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:00.912020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.912020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:00.912020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:00.912020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:00.912688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:00.912688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:00.912688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.912688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:00.912688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:00.912688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:00.912688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:00.912688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:00.912688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:00.912688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:00.912688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:00.912688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:00.912688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:00.912688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:00.912688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:00.912688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:00.912688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:00.912688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:00.912688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:00.913324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:00.913324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:00.913324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:03.799737 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:03.800467 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:03.800467 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:03.800833 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:03.825563 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:03.825826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:03.826420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:03.826420 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:03.826420 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:03.826420 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:03.826420 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:03.826420 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:03.826420 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:03.826420 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:03.827186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:03.827186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:03.827186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:03.827186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:03.827186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:03.827186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:03.827186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:03.827186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:03.827186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:03.827186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:03.827186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:03.827186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:03.828569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:03.828569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:03.828569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:03.828569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:03.828569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:03.828569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:03.828569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:03.828569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:03.828569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:03.828569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:03.828569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:03.828569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:03.828569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:03.828810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:03.828810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:03.828810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:03.828810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:03.828810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:03.828810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:03.828810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:03.828810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:03.828810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:03.828810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:03.828810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:03.828810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:03.828810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:03.828810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:03.828810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:03.829125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:03.829125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:03.829125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:03.829125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:03.829125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:03.829125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:03.829125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:03.829125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:03.829125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:03.829125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:03.829125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:03.829125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:03.829125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:03.829125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:03.829125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:03.829125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:03.829125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:03.829125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:03.829125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:03.829489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:03.829489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:03.829489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:03.829489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:03.829489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:03.829489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:05.877521 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:05.878046 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:05.878046 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:05.878560 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:05.878729 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:05.878837 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:05.878939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:05.879029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.879464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:05.879464 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:05.879464 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:05.879464 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:05.879464 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:05.879464 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:05.879464 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.879464 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:05.880063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.880063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:05.880063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:05.880063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:05.880063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.880063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:05.880063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:05.880063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.880063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:05.880063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:05.880063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:05.880063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.880356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:05.880356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:05.880356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:05.880356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:05.880356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:05.880356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:05.880356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:05.880356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:05.880356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:05.880356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:05.880356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:05.880356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:05.880356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:05.880628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.880628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:05.880805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.880805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:05.880805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:05.880805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:05.880805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:05.880805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:05.880805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.880805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:05.880805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:05.880805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.880805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:05.880805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:05.880805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.880805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:05.880805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:05.880805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:05.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:05.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:05.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:05.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:05.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:05.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:05.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:05.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:05.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:05.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:05.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:05.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:05.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:05.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:05.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:05.881095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:05.881433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:05.881433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.881433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:05.882235 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:05.882445 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:05.882526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:05.882622 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:43:05.883126 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:05.883475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.884072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:05.884072 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:05.884072 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:05.884072 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:05.884072 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:05.884072 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:05.884072 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.884072 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:05.884600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.884600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:05.884600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:05.884600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:05.884600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.884600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:05.884600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:05.884600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:05.884600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.884600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:05.884600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:05.884600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:05.884600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:05.884600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:05.884600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:05.884906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:05.884906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:05.884906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:05.884906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:05.884906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:05.884906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:05.884906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:05.884906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.884906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:05.885139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.885139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:05.885139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:05.885139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:05.885139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.885139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:05.885139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:05.885139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:43:05.885139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:05.885139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:05.885139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.885139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:05.885607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:05.885607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:05.885607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.885607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:05.885607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:05.885607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:05.885607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:05.885607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:05.885607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:05.885607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:05.885607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:05.885607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:05.885607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:05.885607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:05.886034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:05.886034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:05.886034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.886034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:05.886034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.886034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:05.886034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:05.886034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:05.886034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:05.886034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:05.886034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.886034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:05.886034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:05.886034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.886448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:05.886448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:05.886448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.886448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:05.886448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:05.886448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.886448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:05.886448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:05.886448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:43:05.886448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:05.886448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:05.886448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.886448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:05.886448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:05.886448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:05.886448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.886448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:05.886448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:05.886448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:05.886796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:05.886796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:05.886796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:05.886796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:05.886796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:05.886796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:05.886796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:05.886796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:05.886796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:05.886796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:05.886796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.886796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:05.888220 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:05.888494 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:05.888573 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:05.888689 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:05.889109 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:05.889400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.889955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:05.889955 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:05.889955 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:05.889955 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:05.889955 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:05.889955 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:05.889955 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.889955 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:05.890469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.890469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:05.890469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:05.890469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:05.890469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.890469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:05.890469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:05.890469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:05.890469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.890469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:05.890469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:05.890469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:05.890469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:05.890469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:05.890469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:05.890867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:05.890867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:05.890867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:05.890867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:05.890867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:05.890867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:05.890867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:05.890867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.890867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:05.891176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.891176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:05.891176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:05.891176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:05.891176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.891176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:05.891176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:05.891176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:05.891176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:05.891176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.891176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:05.891432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:05.891432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:05.891432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.891432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:05.891432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:05.891432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:05.891432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:05.891432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:05.891432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:05.891432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:05.891432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:05.891432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:05.891432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:05.891432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:05.891676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:05.891676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:05.891676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.891676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:05.891853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.891853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:05.891853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:05.891853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:05.891853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:05.891853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:05.891853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.891853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:05.891853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:05.891853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.891853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:05.891853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:05.891853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.891853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:05.891853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:05.891853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.891853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:05.892166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:05.892166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:05.892166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:05.892166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.892166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:05.892166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:05.892166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:05.892166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.892166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:05.892166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:05.892166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:05.892166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:05.892166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:05.892166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:05.892166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:05.892166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:05.892166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:05.892166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:05.892166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:05.892467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:05.892467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:05.892467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.892467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:05.908912 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:05.909341 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:05.909435 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:05.909588 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:05.909880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:05.910069 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:05.910209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.910652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:05.910652 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:05.910652 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:05.910652 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:05.910652 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:05.910652 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:05.910652 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.910652 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:05.911323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.911323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:05.911323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:05.911323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:05.911323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.911323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:05.911323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:05.911323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.911323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:05.911323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:05.911323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:05.911323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.911567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:05.911567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:05.911567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:05.911567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:05.911567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:05.911567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:05.911567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:05.911567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:05.911567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:05.911567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:05.911567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:05.911567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:05.911567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:05.911777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.911777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:05.912002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.912002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:05.912002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:05.912002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:05.912002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:05.912002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:05.912002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.912002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:05.912002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:05.912002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.912002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:05.912002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:05.912002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.912002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:05.912002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:05.912002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:05.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:05.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:05.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:05.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:05.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:05.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:05.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:05.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:05.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:05.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:05.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:05.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:05.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:05.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:05.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:05.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:05.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:05.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:05.912640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:05.912640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:05.912640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:08.821382 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:08.821382 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:08.821382 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:08.821382 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:12.702349 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:12.702349 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:12.702349 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:12.702349 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:12.702349 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:12.702349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.702349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:12.730096 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:12.730096 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:12.730096 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:12.730096 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:12.730096 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:12.730096 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.730096 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:12.730096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 3.656 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.730096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:12.730096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:12.730096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:12.730096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.730096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:12.730096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:12.730096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.730096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:12.735455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:12.735455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:12.735455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.735455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:12.735455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:12.735455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:12.735455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:12.735455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:12.735455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:12.735455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:12.735455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:12.735455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:12.735455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:12.735455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:12.736867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:12.736867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:12.736867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.736867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:12.736867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.736867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:12.736867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:12.736867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:12.736867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:12.736867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:12.736867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.736867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:12.736867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:12.736867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.737744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:12.737744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:12.737744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.737744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:12.737744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:12.737744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.737744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:12.737744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:12.737744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.737744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:12.737744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:12.737744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:12.737744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.737744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:12.737744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:12.737744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:12.737744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:12.737744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:12.737744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:12.738580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:12.738580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:12.738580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:12.738580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:12.738580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:12.738580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:12.738580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:12.738580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.738580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:12.738580 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:12.738580 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:12.738580 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:12.739134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:12.739134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:12.739134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:12.739134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.739134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:12.739134 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:12.739134 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:12.739134 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:12.739134 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:12.739134 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:12.739134 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.739134 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:12.740692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.740692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:12.740692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:12.740692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:12.740692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.740692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:12.740692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:12.740692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.740692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:12.740692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:12.740692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:12.740692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.742032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:12.742032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:12.742032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:12.742032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:12.742032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:12.742032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:12.742032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:12.742032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:12.742032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:12.742032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:12.742032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:12.742032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:12.742032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:12.742634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.742634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:12.742634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.742634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:12.742634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:12.742634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:12.742634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:12.742634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:12.742634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.742634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:12.742634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:12.742634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.742634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:12.742634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:12.742634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.743585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:12.743585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:12.743585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.743585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:12.743585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:12.743585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.743585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:12.743585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:12.743585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:12.743585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.743585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:12.743585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:12.743585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:12.743585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:12.743585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:12.743585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:12.743585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:12.743585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:12.743585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:12.744751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:12.744751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:12.744751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:12.744751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:12.744751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.744751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:12.744751 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:12.744751 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:12.744751 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:43:12.744751 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:12.745360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.745360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:12.745360 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:12.745360 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:12.745360 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:12.745360 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:12.745360 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:12.745360 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.745360 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:12.745360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.745360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:12.745360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:12.745360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:12.745360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.745360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:12.745360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:12.745360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:12.746915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.746915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:12.746915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:12.746915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:12.746915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:12.746915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:12.746915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:12.746915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:12.746915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:12.746915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:12.746915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:12.746915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:12.746915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:12.746915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:12.746915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.746915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:12.747565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.747565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:12.747565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:12.747565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:12.747565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.747565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:12.747565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:12.747565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:43:12.747565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:12.747565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:12.747565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.747565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:12.748778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:12.748778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:12.748778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.748778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:12.748778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:12.748778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:12.748778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:12.748778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:12.748778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:12.748778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:12.748778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:12.748778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:12.748778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:12.748778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:12.749330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:12.749330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:12.749330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.749330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:12.749330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.749330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:12.749330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:12.749330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:12.749330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:12.749330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:12.749330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.749330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:12.749330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:12.749330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.749826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:12.749826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:12.749826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.749826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:12.749826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:12.749826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.749826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:12.749826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:12.749826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:43:12.749826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:12.749826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:12.749826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.749826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:12.749826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:12.749826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:12.749826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.749826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:12.749826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:12.749826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:12.750778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:12.750778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:12.750778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:12.750778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:12.750778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:12.750778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:12.750778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:12.750778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:12.750778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:12.750778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:12.750778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.750778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:12.750778 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:12.750778 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:12.751866 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:12.751866 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:12.751866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.751866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:12.751866 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:12.751866 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:12.751866 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:12.751866 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:12.751866 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:12.751866 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.751866 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:12.751866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.752695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:12.752695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:12.752695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:12.752695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.752695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:12.752695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:12.752695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:12.752695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.752695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:12.752695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:12.752695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:12.752695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:12.752695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:12.752695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:12.752695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:12.752695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:12.752695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:12.753240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:12.753240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:12.753240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:12.753240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:12.753240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.753240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:12.753240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.753240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:12.753240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:12.753240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:12.753240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.753240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:12.753240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:12.754372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:12.754372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:12.754372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.754372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:12.754372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:12.754372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:12.754372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.754372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:12.754372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:12.754372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:12.754372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:12.754372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:12.754372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:12.755328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:12.755328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:12.755328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:12.755328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:12.755328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:12.755328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:12.755328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:12.755328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.755328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:12.755328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.755328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:12.755328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:12.756061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:12.756061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:12.756061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:12.756061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.756061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:12.756061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:12.756061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.756061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:12.756061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:12.756061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.756061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:12.756061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:12.756061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.756061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:12.756061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:12.756061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:12.756061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:12.756061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.756061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:12.757439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:12.757439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:12.757439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.757439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:12.757439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:12.757439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:12.757439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:12.757439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:12.757439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:12.757439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:12.757439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:12.757439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:12.757439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:12.757439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:12.757439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:12.757439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:12.757439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.757439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:12.757955 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:12.757955 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:12.757955 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:12.757955 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:12.757955 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:12.757955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.758454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:12.758454 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:12.758454 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:12.758454 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:12.758454 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:12.758454 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:12.758454 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.758454 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:12.758454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.758454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:12.758454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:12.758454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:12.758454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.758454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:12.758454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:12.759844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.759844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:12.759844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:12.759844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:12.759844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.759844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:12.759844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:12.759844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:12.759844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:12.759844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:12.759844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:12.759844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:12.759844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:12.759844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:12.760492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:12.760492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:12.760492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:12.760492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:12.760492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.760492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:12.760492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.760492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:12.760492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:12.760492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:12.760492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:12.760492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:12.760492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.760492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:12.761572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:12.761965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:12.761965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:12.761965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:12.761965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:12.761965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:12.761965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:12.761965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:12.761965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:12.761965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:12.761965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:13.823609 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:13.824424 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:13.824424 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:13.824880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:13.849935 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:13.850182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:13.850684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:13.850684 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:13.850684 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:13.850684 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:13.850684 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:13.850684 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:13.850684 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:13.850684 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:13.851307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:13.851307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:13.851307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:13.851307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:13.851307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:13.851307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:13.851307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:13.851307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:13.851307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:13.851307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:13.851307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:13.851307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:13.852650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:13.852650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:13.852650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:13.852650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:13.852650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:13.852650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:13.852650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:13.852650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:13.852650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:13.852650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:13.852650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:13.852650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:13.852650 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:13.853130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:13.853130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:13.853130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:13.853130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:13.853130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:13.853130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:13.853130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:13.853130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:13.853130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:13.853130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:13.853130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:13.853130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:13.853130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:13.853130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:13.853130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:13.853634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:13.853634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:13.853634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:13.853634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:13.853634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:13.853634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:13.853634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:13.853634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:13.853634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:13.853634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:13.853634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:13.853634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:13.853634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:13.853634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:13.853634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:13.853634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:13.853634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:13.853634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:13.853634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:13.854242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:13.854242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:13.854242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:13.854242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:13.854242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:13.854242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:17.619497 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:17.619922 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:17.619922 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:17.620517 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:17.620743 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:17.620843 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:17.620950 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:17.621058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.621500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:17.621500 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:17.621500 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:17.621500 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:17.621500 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:17.621500 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:17.621500 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.621500 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:17.622228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.622228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:17.622228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:17.622228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:17.622228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.622228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:17.622228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:17.622228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.622228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:17.622228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:17.622228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:17.622228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.622509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:17.622509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:17.622509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:17.622509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:17.622509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:17.622509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:17.622509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:17.622509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:17.622509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:17.622509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:17.622509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:17.622509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:17.622509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:17.622737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.622737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:17.622967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.622967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:17.622967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:17.622967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:17.622967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:17.622967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:17.622967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.622967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:17.622967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:17.622967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.622967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:17.622967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:17.622967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.622967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:17.622967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:17.622967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.623240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:17.623240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:17.623240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.623240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:17.623240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:17.623240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:17.623240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.623240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:17.623240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:17.623240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:17.623240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:17.623240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:17.623240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:17.623240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:17.623240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:17.623240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:17.623240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:17.623240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:17.623240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:17.623615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:17.623615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.623615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:17.623978 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:17.624121 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:17.624200 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:17.624302 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:43:17.624563 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:17.624717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:17.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:17.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:17.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:17.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:17.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:17.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:17.625586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.625586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:17.625586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:17.625586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:17.625586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.625586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:17.625586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:17.625586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:17.625586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.625586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:17.625586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:17.625586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:17.625586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:17.625586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:17.625586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:17.625875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:17.625875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:17.625875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:17.625875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:17.625875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:17.625875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:17.625875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:17.625875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.625875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:17.626149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.626149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:17.626149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:17.626149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:17.626149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.626149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:17.626149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:17.626149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:43:17.626149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:17.626149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:17.626149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.626149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:17.626774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:17.626774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:17.626774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.626774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:17.626774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:17.626774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:17.626774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:17.626774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:17.626774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:17.626774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:17.626774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:17.626774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:17.626774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:17.626774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:17.626992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:17.626992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:17.626992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.626992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:17.626992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.626992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:17.626992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:17.626992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:17.626992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:17.626992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:17.626992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.626992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:17.626992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:17.626992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.627212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:17.627212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:17.627212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.627212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:17.627212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:17.627212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.627212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:17.627212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:17.627212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:43:17.627212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:17.627212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:17.627212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.627212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:17.627212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:17.627212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:17.627212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.627212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:17.627212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:17.627212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:17.627575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:17.627575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:17.627575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:17.627575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:17.627575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:17.627575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:17.627575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:17.627575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:17.627575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:17.627575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:17.627575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.627575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:17.628446 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:17.628618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:17.628660 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:17.628734 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:17.629061 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:17.629295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.629695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:17.629695 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:17.629695 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:17.629695 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:17.629695 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:17.629695 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:17.629695 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.629695 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:17.630230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.630230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:17.630230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:17.630230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:17.630230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.630230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:17.630230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:17.630230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:17.630230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.630230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:17.630230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:17.630230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:17.630230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:17.630230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:17.630230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:17.630685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:17.630685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:17.630685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:17.630685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:17.630685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:17.630685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:17.630685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:17.630685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.630685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:17.630891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.630891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:17.630891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:17.630891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:17.630891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.630891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:17.630891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:17.630891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:17.630891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:17.630891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.630891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:17.631101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:17.631101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:17.631101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.631101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:17.631101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:17.631101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:17.631101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:17.631101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:17.631101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:17.631101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:17.631101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:17.631101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:17.631101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:17.631101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:17.631410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:17.631410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:17.631410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.631410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:17.631560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.631560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:17.631560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:17.631560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:17.631560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:17.631560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:17.631560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.631560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:17.631560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:17.631560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.631560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:17.631560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:17.631560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.631560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:17.631560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:17.631560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.631560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:17.631887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:17.631887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:17.631887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:17.631887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.631887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:17.631887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:17.631887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:17.631887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.631887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:17.631887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:17.631887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:17.631887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:17.631887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:17.631887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:17.631887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:17.631887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:17.631887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:17.631887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:17.631887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:17.632218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:17.632218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:17.632218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.632218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:17.633424 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:17.633596 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:17.633689 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:17.633825 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:17.634064 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:17.634228 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:17.634380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.634784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:17.634784 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:17.634784 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:17.634784 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:17.634784 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:17.634784 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:17.634784 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.634784 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:17.635415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.635415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:17.635415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:17.635415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:17.635415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.635415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:17.635415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:17.635415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.635415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:17.635415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:17.635415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:17.635415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.635683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:17.635683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:17.635683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:17.635683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:17.635683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:17.635683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:17.635683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:17.635683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:17.635683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:17.635683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:17.635683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:17.635683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:17.635683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:17.636002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.636002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:17.636086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.636086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:17.636086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:17.636086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:17.636086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:17.636086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:17.636086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.636086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:17.636086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:17.636086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.636086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:17.636086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:17.636086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.636086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:17.636086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:17.636086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:17.636434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:17.636434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:17.636434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.636434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:17.636434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:17.636434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:17.636434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:17.636434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:17.636434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:17.636434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:17.636434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:17.636434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:17.636434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:17.636434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:17.636434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:17.636434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:17.636434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:17.636434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:17.636434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:17.636832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:17.636832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:17.636832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:18.824018 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:18.824369 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:18.824406 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:18.825030 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:18.857903 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:18.858102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:18.858643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:18.858643 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:18.858643 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:18.858643 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:18.858643 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:18.858643 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:18.858643 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:18.858643 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:18.859339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:18.859339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:18.859339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:18.859339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:18.859339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:18.859339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:18.859339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:18.859339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:18.859339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:18.859339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:18.859339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:18.859339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:18.860210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:18.860210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:18.860210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:18.860210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:18.860210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:18.860210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:18.860210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:18.860210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:18.860210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:18.860210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:18.860210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:18.860210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:18.860210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:18.860436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:18.860436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:18.860436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:18.860436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:18.860436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:18.860436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:18.860436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:18.860436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:18.860436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:18.860436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:18.860436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:18.860436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:18.860436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:18.860436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:18.860436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:18.860723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:18.860723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:18.860723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:18.860723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:18.860723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:18.860723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:18.860723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:18.860723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:18.860723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:18.860723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:18.860723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:18.860723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:18.860723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:18.860723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:18.860723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:18.860723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:18.860723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:18.860723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:18.860723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:18.861059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:18.861059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:18.861059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:18.861059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:18.861059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:18.861059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:22.619915 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:22.620341 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:22.620341 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:22.621004 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:22.621265 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:22.621417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:22.621532 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:22.621673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.622193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:22.622193 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:22.622193 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:22.622193 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:22.622193 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:22.622193 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:22.622193 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.622193 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:22.623085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.623085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:22.623085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:22.623085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:22.623085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.623085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:22.623085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:22.623085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.623085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:22.623085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:22.623085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:22.623085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.623328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:22.623328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:22.623328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:22.623328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:22.623328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:22.623328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:22.623328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:22.623328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:22.623328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:22.623328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:22.623328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:22.623328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:22.623328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:22.623533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.623533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:22.624018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.624018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:22.624018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:22.624018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:22.624018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:22.624018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:22.624018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.624018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:22.624018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:22.624018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.624018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:22.624018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:22.624018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.624018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:22.624018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:22.624018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.624317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:22.624317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:22.624317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.624317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:22.624317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:22.624317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:22.624317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.624317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:22.624317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:22.624317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:22.624317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:22.624317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:22.624317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:22.624317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:22.624317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:22.624317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:22.624317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:22.624317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:22.624317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:22.624637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:22.624637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.624637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:22.624637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:22.625295 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:22.625295 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:22.625295 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:43:22.625520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:22.625685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.627961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:22.627961 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:22.627961 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:22.627961 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:22.627961 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:22.627961 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:22.627961 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.627961 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:22.627961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.627961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:22.627961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:22.627961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:22.627961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.627961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:22.627961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:22.627961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:22.627961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.627961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:22.628351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:22.628351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:22.628351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:22.628351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:22.628351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:22.628351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:22.628351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:22.628351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:22.628351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:22.628351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:22.628351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:22.628351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:22.628351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.628351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:22.628351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.628642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:22.628642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:22.628642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:22.628642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.628642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:22.628642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:22.628642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:43:22.628642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:22.628642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:22.628642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.628642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:22.628642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:22.628642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:22.628862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.628862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:22.628862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:22.628862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:22.628862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:22.628862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:22.628862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:22.628862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:22.628862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:22.628862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:22.628862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:22.628862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:22.628862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:22.628862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:22.629146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.629146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:22.629146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.629146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:22.629146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:22.629146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:22.629146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:22.629146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:22.629146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.629146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:22.629146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:22.629146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.629146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:22.629146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:22.629146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.629420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:22.629420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:22.629420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.629420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:22.629420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:22.629420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:43:22.629420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:22.629420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:22.629420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.629420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:22.629420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:22.629420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:22.629420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.629420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:22.629420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:22.629420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:22.629420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:22.629420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:22.629420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:22.629751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:22.629751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:22.629751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:22.629751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:22.629751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:22.629751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:22.629751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:22.629751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.629751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:22.629751 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:22.629751 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:22.629751 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:22.629975 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:22.629975 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:22.630145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.630500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:22.630500 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:22.630500 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:22.630500 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:22.630500 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:22.630500 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:22.630500 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.630500 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:22.631027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.631027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:22.631027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:22.631027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:22.631027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.631027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:22.631027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:22.631027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:22.631027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.631027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:22.631027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:22.631027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:22.631027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:22.631027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:22.631027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:22.631314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:22.631314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:22.631314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:22.631314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:22.631314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:22.631314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:22.631314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:22.631314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.631314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:22.631641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.631641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:22.631641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:22.631641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:22.631641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.631641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:22.631641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:22.631641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:22.631641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:22.631641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.631641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:22.632264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:22.632264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:22.632264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.632264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:22.632264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:22.632264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:22.632264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:22.632264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:22.632264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:22.632264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:22.632264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:22.632264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:22.632264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:22.632264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:22.632497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:22.632497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:22.632497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.632497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:22.632497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.632497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:22.632497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:22.632497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:22.632497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:22.632497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:22.632497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.632497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:22.632497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:22.632497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.632759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:22.632759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:22.632759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.632759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:22.632759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:22.632759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.632759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:22.632759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:22.632759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:22.632759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:22.632759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.632759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:22.632759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:22.632759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:22.632759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.632759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:22.632759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:22.632759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:22.632759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:22.633079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:22.633079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:22.633079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:22.633079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:22.633079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:22.633079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:22.633079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:22.633079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:22.633079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:22.633079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.633079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:22.633729 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:22.633936 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:22.634028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:22.634148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:22.634372 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:22.634530 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:22.634665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.634987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:22.634987 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:22.634987 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:22.634987 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:22.634987 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:22.634987 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:22.634987 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.634987 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:22.635519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.635519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:22.635519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:22.635519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:22.635519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.635519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:22.635519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:22.635519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.635519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:22.635519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:22.635519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:22.635519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.635765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:22.635765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:22.635765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:22.635765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:22.635765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:22.635765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:22.635765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:22.635765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:22.635765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:22.635765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:22.635765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:22.635765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:22.635765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:22.635981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.635981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:22.636170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.636170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:22.636170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:22.636170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:22.636170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:22.636170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:22.636170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.636170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:22.636170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:22.636170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.636170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:22.636170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:22.636170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.636170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:22.636170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:22.636170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:22.636473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:22.636473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:22.636473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.636473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:22.636473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:22.636473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:22.636473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:22.636473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:22.636473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:22.636473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:22.636473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:22.636473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:22.636473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:22.636473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:22.636473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:22.636473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:22.636473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:22.636473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:22.636473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:22.636812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:22.636812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:22.636812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:23.824508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:23.825108 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:23.825208 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:23.825852 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:23.854768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:23.855003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:23.855371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:23.855371 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:23.855371 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:23.855371 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:23.855371 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:23.855371 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:23.855371 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:23.855371 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:23.856001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:23.856001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:23.856001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:23.856001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:23.856001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:23.856001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:23.856001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:23.856001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:23.856001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:23.856001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:23.856001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:23.856001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:23.856331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:23.856331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:23.856331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:23.856331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:23.856331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:23.856331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:23.856331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:23.856331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:23.856331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:23.856331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:23.856331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:23.856331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:23.856331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:23.856718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:23.856718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:23.856889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:23.856889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:23.856889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:23.856889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:23.856889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:23.856889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:23.856889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:23.856889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:23.856889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:23.856889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:23.856889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:23.856889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:23.856889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:23.856889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:23.856889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:23.856889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:23.857328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:23.857328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:23.857328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:23.857328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:23.857328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:23.857328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:23.857328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:23.857328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:23.857328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:23.857328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:23.857328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:23.857328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:23.857328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:23.857328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:23.857328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:23.857328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:23.857328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:23.857328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:23.857328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:23.858208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:23.858208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:23.858208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:27.620342 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:27.620775 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:27.620775 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:27.621444 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:27.621629 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:27.621737 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:27.621842 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:27.621945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.622406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:27.622406 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:27.622406 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:27.622406 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:27.622406 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:27.622406 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:27.622406 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.622406 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:27.623101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.623101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:27.623101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:27.623101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:27.623101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.623101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:27.623101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:27.623101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.623101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:27.623101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:27.623101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:27.623101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.623459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:27.623459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:27.623459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:27.623459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:27.623459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:27.623459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:27.623459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:27.623459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:27.623459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:27.623459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:27.623459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:27.623459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:27.623459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:27.623915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.623915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:27.624030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.624030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:27.624030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:27.624030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:27.624030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:27.624030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:27.624030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.624030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:27.624030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:27.624030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.624030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:27.624030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:27.624030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.624030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:27.624030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:27.624030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.624466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:27.624466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:27.624466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.624466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:27.624466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:27.624466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:27.624466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.624466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:27.624466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:27.624466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:27.624466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:27.624466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:27.624466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:27.624466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:27.624466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:27.624466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:27.624466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:27.624466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:27.624466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:27.624916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:27.624916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.624916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:27.624995 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:27.625438 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:27.625438 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:27.625438 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:43:27.625715 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:27.625919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.626432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:27.626432 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:27.626432 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:27.626432 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:27.626432 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:27.626432 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:27.626432 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.626432 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:27.626980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.626980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:27.626980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:27.626980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:27.626980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.626980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:27.626980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:27.626980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:27.626980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.626980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:27.626980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:27.626980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:27.626980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:27.626980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:27.626980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:27.627808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:27.627808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:27.627808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:27.627808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:27.627808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:27.627808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:27.627808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:27.627808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.627808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:27.627808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.627808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:27.627808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:27.627808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:27.627808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.628563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:27.628563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:27.628563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:43:27.628563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:27.628563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:27.628563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.628563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:27.628563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:27.628563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:27.628563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.628563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:27.628563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:27.628563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:27.628997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:27.628997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:27.628997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:27.628997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:27.628997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:27.628997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:27.628997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:27.628997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:27.628997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:27.628997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:27.628997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.628997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:27.629292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.629292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:27.629292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:27.629292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:27.629292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:27.629292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:27.629292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.629292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:27.629292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:27.629292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.629292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:27.629292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:27.629292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.629292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:27.629292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:27.629292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.629672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:27.629672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:27.629672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:43:27.629672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:27.629672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:27.629672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.629672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:27.629672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:27.629672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:27.629672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.629672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:27.629672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:27.629672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:27.629672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:27.629672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:27.629672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:27.629672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:27.629672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:27.630085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:27.630085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:27.630085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:27.630085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:27.630085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:27.630085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.630085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:27.630085 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:27.630085 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:27.630085 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:27.630367 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:27.630367 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:27.630367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.630756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:27.630756 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:27.630756 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:27.630756 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:27.630756 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:27.630756 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:27.630756 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.630756 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:27.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:27.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:27.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:27.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:27.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:27.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:27.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:27.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:27.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:27.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:27.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:27.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:27.631652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:27.631652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:27.631652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:27.631652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:27.631652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:27.631652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:27.631652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:27.631652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.631652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:27.631876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.631876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:27.631876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:27.631876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:27.631876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.631876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:27.631876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:27.631876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:27.631876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:27.631876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.631876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:27.632167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:27.632167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:27.632167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.632167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:27.632167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:27.632167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:27.632167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:27.632167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:27.632167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:27.632167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:27.632167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:27.632167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:27.632167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:27.632167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:27.632539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:27.632539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:27.632539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.632539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:27.632539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.632539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:27.632539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:27.632539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:27.632539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:27.632539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:27.632539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.632539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:27.632539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:27.632539 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.632908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:27.632908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:27.632908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.632908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:27.632908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:27.632908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.632908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:27.632908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:27.632908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:27.632908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:27.632908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.632908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:27.632908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:27.632908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:27.632908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.632908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:27.632908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:27.632908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:27.632908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:27.633399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:27.633399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:27.633399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:27.633399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:27.633399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:27.633399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:27.633399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:27.633399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:27.633399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:27.633399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.633399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:27.633991 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:27.634183 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:27.634281 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:27.634398 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:27.634661 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:27.634827 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:27.634952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.635304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:27.635304 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:27.635304 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:27.635304 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:27.635304 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:27.635304 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:27.635304 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.635304 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:27.635847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.635847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:27.635847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:27.635847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:27.635847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.635847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:27.635847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:27.635847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.635847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:27.635847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:27.635847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:27.635847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.636178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:27.636178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:27.636178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:27.636178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:27.636178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:27.636178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:27.636178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:27.636178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:27.636178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:27.636178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:27.636178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:27.636178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:27.636178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:27.636641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.636641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:27.636641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.636641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:27.636641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:27.636641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:27.636641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:27.636641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:27.636641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.636641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:27.636641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:27.636641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.636641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:27.636641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:27.636641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.637042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:27.637042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:27.637042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:27.637042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:27.637042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:27.637042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.637042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:27.637042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:27.637042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:27.637042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:27.637042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:27.637042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:27.637042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:27.637042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:27.637042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:27.637042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:27.637042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:27.637042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:27.637042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:27.637523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:27.637523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:27.637523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:27.637523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:27.637523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:27.637523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:28.825238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:28.825702 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:28.825781 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:28.826404 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:28.851561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:28.851731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:28.852176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:28.852176 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:28.852176 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:28.852176 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:28.852176 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:28.852176 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:28.852176 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:28.852176 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:28.852882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:28.852882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:28.852882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:28.852882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:28.852882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:28.852882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:28.852882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:28.852882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:28.852882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:28.852882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:28.852882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:28.852882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:28.853268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:28.853268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:28.853268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:28.853268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:28.853268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:28.853268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:28.853268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:28.853268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:28.853268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:28.853268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:28.853268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:28.853268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:28.853268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:28.853580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:28.853580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:28.853895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:28.853895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:28.853895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:28.853895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:28.853895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:28.853895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:28.853895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:28.853895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:28.853895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:28.853895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:28.853895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:28.853895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:28.853895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:28.853895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:28.853895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:28.853895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:28.854169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:28.854169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:28.854169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:28.854169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:28.854169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:28.854169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:28.854169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:28.854169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:28.854169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:28.854169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:28.854169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:28.854169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:28.854169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:28.854169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:28.854169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:28.854169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:28.854169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:28.854169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:28.854169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:28.854505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:28.854505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:28.854505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:32.620798 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:32.621833 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:32.621833 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:32.622230 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:32.622474 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:32.622719 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:32.622827 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:32.622937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.624784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:32.624784 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:32.624784 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:32.624784 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:32.624784 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:32.624784 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:32.624784 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.624784 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:32.624784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.624784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:32.624784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:32.624784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:32.624784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.624784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:32.624784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:32.625056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.625056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:32.625056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:32.625056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:32.625056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.625056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:32.625056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:32.625056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:32.625056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:32.625056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:32.625056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:32.625056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:32.625056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:32.625056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:32.625420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:32.625420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:32.625420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:32.625420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:32.625420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.625420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:32.625420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.625420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:32.625420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:32.625420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:32.625420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:32.625420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:32.625420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.625420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:32.625699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:32.626031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:32.626031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:32.626031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:32.626031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:32.626031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:32.626031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:32.626031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:32.626031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:32.626031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.626031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:32.626031 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:32.626031 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:32.626031 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:32.626268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:43:32.626268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:32.626385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.626791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:32.626791 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:32.626791 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:32.626791 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:32.626791 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:32.626791 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:32.626791 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.626791 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:32.627629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.627629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:32.627629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:32.627629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:32.627629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.627629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:32.627629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:32.627629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:32.627629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.627629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:32.627629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:32.627629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:32.627629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:32.627629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:32.627629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:32.627922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:32.627922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:32.627922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:32.627922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:32.627922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:32.627922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:32.627922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:32.627922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.627922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:32.628587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.628587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:32.628587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:32.628587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:32.628587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.628587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:32.628587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:32.628587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:43:32.628587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:32.628587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:32.628587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.628587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:32.628806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:32.628806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:32.628806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.628806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:32.628806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:32.628806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:32.628806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:32.628806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:32.628806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:32.628806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:32.628806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:32.628806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:32.628806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:32.628806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:32.629048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:32.629048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:32.629048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.629048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:32.629831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.629831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:32.629831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:32.629831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:32.629831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:32.629831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:32.629831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.629831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:32.629831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:32.629831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.629831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:32.629831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:32.629831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.629831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:32.629831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:32.629831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.630118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:32.630118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:32.630118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:43:32.630118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:32.630118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:32.630118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.630118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:32.630118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:32.630118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:32.630118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.630118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:32.630118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:32.630118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:32.630118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:32.630118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:32.630118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:32.630118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:32.630118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:32.630429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:32.630429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:32.630429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:32.630429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:32.630429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:32.630429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.630429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:32.630429 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:32.630429 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:32.630429 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:32.630661 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:32.630828 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:32.630990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.631345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:32.631345 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:32.631345 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:32.631345 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:32.631345 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:32.631345 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:32.631345 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.631345 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:32.632116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.632116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:32.632116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:32.632116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:32.632116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.632116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:32.632116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:32.632116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:32.632116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.632116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:32.632116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:32.632116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:32.632116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:32.632116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:32.632116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:32.632396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:32.632396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:32.632396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:32.632396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:32.632396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:32.632396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:32.632396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:32.632396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.632396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:32.633057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.633057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:32.633057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:32.633057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:32.633057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.633057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:32.633057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:32.633057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:32.633057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:32.633057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.633057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:32.633294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:32.633294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:32.633294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.633294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:32.633294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:32.633294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:32.633294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:32.633294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:32.633294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:32.633294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:32.633294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:32.633294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:32.633294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:32.633294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:32.633528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:32.633528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:32.633528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.633528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:32.634297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.634297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:32.634297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:32.634297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:32.634297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:32.634297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:32.634297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.634297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:32.634297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:32.634297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.634297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:32.634297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:32.634297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.634297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:32.634297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:32.634297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.634297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:32.634598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:32.634598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:32.634598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:32.634598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.634598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:32.634598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:32.634598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:32.634598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.634598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:32.634598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:32.634598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:32.634598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:32.634598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:32.634598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:32.634598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:32.634598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:32.634598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:32.634598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:32.634598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:32.634908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:32.634908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:32.634908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.634908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:32.634908 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:32.634908 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:32.634908 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:32.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:32.635302 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:32.635459 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:32.635605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.635938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:32.635938 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:32.635938 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:32.635938 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:32.635938 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:32.635938 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:32.635938 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.635938 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:32.636845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.636845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:32.636845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:32.636845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:32.636845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.636845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:32.636845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:32.636845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.636845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:32.636845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:32.636845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:32.636845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.637061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:32.637061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:32.637061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:32.637061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:32.637061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:32.637061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:32.637061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:32.637061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:32.637061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:32.637061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:32.637061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:32.637061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:32.637061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:32.637304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.637304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:32.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:32.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:32.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:32.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:32.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:32.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:32.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:32.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:32.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:32.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:32.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:32.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:32.638241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:32.638241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:32.638241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.638241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:32.638241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:32.638241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:32.638241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:32.638241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:32.638241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:32.638241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:32.638241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:32.638241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:32.638241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:32.638241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:32.638241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:32.638241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:32.638241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:32.638241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:32.638241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:32.638567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:32.638567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:32.638567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:33.825727 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:33.826153 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:33.826194 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:33.826828 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:33.857293 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:33.857293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:33.857293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:33.857293 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:33.857293 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:33.857293 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:33.857293 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:33.857293 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:33.857293 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:33.857293 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:33.857293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:33.857293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:33.857293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:33.857705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:33.857705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:33.857705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:33.857705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:33.857705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:33.857705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:33.857705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:33.857705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:33.857705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:33.857705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:33.857705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:33.857705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:33.857705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:33.857705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:33.858001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:33.858001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:33.858001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:33.858001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:33.858001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:33.858001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:33.858001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:33.858001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:33.858001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:33.858001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:33.858001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:33.858001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:33.858266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:33.858266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:33.858266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:33.858266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:33.858266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:33.858266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:33.858266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:33.858266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:33.858266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:33.858266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:33.858266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:33.858266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:33.858266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:33.858266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:33.858266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:33.858266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:33.858266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:33.858266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:33.858266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:33.858641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:33.858641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:33.858641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:33.858641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:33.858641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:33.858641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:33.858641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:33.858641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:33.858641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:33.858641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:33.858641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:33.858641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:33.858641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:33.858641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:33.858641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:33.858641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:33.858641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:37.621591 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:37.622091 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:37.622192 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:37.623032 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:37.623184 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:37.623326 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:37.623432 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:37.623533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.624057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:37.624057 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:37.624057 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:37.624057 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:37.624057 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:37.624057 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:37.624057 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.624057 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:37.624725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.624725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:37.624725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:37.624725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:37.624725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.624725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:37.624725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:37.624725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.624725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:37.624725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:37.624725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:37.624725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:37.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:37.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:37.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:37.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:37.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:37.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:37.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:37.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:37.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:37.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:37.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:37.625058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:37.625321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.625321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:37.625531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.625531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:37.625531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:37.625531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:37.625531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:37.625531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:37.625531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.625531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:37.625531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:37.625531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.625531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:37.625531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:37.625531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.625531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:37.625531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:37.625531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.626165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:37.626165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:37.626165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.626165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:37.626165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:37.626165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:37.626165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.626165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:37.626165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:37.626165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:37.626165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:37.626165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:37.626165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:37.626165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:37.626165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:37.626165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:37.626165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:37.626165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:37.626165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:37.626487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:37.626487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.626487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:37.626487 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:37.626487 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:37.626487 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:37.626487 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:43:37.626487 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:37.627001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.627001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:37.627001 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:37.627001 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:37.627001 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:37.627001 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:37.627001 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:37.627001 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.627001 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:37.627509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.627509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:37.627509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:37.627509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:37.627509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.627509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:37.627509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:37.627509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:37.627509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.627509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:37.627509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:37.627509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:37.627509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:37.627509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:37.627509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:37.627925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:37.627925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:37.627925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:37.627925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:37.627925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:37.627925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:37.627925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:37.627925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.627925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:37.628133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.628133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:37.628133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:37.628133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:37.628133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.628133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:37.628133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:37.628133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:43:37.628133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:37.628133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:37.628133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.628133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:37.628392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:37.628392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:37.628392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.628392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:37.628392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:37.628392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:37.628392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:37.628392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:37.628392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:37.628392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:37.628392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:37.628392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:37.628392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:37.628392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:37.628679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:37.628679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:37.628679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.628679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:37.628903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.628903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:37.628903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:37.628903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:37.628903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:37.628903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:37.628903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.628903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:37.628903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:37.628903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.628903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:37.628903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:37.628903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.628903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:37.628903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:37.628903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.629264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:37.629264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:37.629264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:43:37.629264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:37.629264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:37.629264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.629264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:37.629264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:37.629264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:37.629264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.629264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:37.629264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:37.629264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:37.629264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:37.629264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:37.629264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:37.629264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:37.629264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:37.630053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:37.630053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:37.630053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:37.630053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:37.630053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:37.630053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.630053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:37.630282 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:37.630417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:37.630497 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:37.630618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:37.630867 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:37.631059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.631899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:37.631899 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:37.631899 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:37.631899 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:37.631899 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:37.631899 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:37.631899 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.631899 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:37.632594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.632594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:37.632594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:37.632594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:37.632594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.632594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:37.632594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:37.632594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:37.632594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.632594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:37.632594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:37.632594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:37.632594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:37.632594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:37.632594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:37.632929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:37.632929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:37.632929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:37.632929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:37.632929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:37.632929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:37.632929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:37.632929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.632929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:37.633246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.633246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:37.633246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:37.633246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:37.633246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.633246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:37.633246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:37.633246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:37.633246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:37.633246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.633246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:37.633510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:37.633510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:37.633510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.633510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:37.633510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:37.633510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:37.633510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:37.633510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:37.633510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:37.633510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:37.633510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:37.633510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:37.633510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:37.633510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:37.633749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:37.633749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:37.633749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.633749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:37.634068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.634068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:37.634068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:37.634068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:37.634068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:37.634068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:37.634068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.634068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:37.634068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:37.634068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.634068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:37.634068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:37.634068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.634068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:37.634068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:37.634068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.634068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:37.634658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:37.634658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:37.634658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:37.634658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.634658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:37.634658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:37.634658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:37.634658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.634658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:37.634658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:37.634658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:37.634658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:37.634658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:37.634658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:37.634658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:37.634658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:37.634658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:37.634658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:37.634658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:37.634952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:37.634952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:37.634952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.634952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:37.634952 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:37.635056 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:37.635166 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:37.635262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:37.635513 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:37.635760 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:37.635898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.636268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:37.636268 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:37.636268 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:37.636268 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:37.636268 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:37.636268 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:37.636268 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.636268 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:37.636854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.636854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:37.636854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:37.636854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:37.636854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.636854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:37.636854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:37.636854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.636854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:37.636854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:37.636854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:37.636854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:37.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:37.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:37.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:37.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:37.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:37.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:37.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:37.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:37.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:37.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:37.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:37.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:37.637318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.637318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:37.637585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.637585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:37.637585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:37.637585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:37.637585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:37.637585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:37.637585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.637585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:37.637585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:37.637585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.637585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:37.637585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:37.637585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.637585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:37.637585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:37.637585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:37.637985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:37.637985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:37.637985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.637985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:37.637985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:37.637985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:37.637985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:37.637985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:37.637985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:37.637985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:37.637985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:37.637985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:37.637985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:37.637985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:37.637985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:37.637985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:37.637985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:37.637985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:37.637985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:37.638608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:37.638608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:37.638608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:38.826197 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:38.826815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:38.826815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:38.827437 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:38.862266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:38.862460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:38.863111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:38.863111 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:38.863111 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:38.863111 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:38.863111 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:38.863111 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:38.863111 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:38.863111 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:38.864331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:38.864331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:38.864331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:38.864331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:38.864331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:38.864331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:38.864331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:38.864331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:38.864331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:38.864331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:38.864331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:38.864331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:38.864622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:38.864622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:38.864622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:38.864622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:38.864622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:38.864622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:38.864622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:38.864622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:38.864622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:38.864622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:38.864622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:38.864622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:38.864622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:38.864891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:38.864891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:38.865871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:38.865871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:38.865871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:38.865871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:38.865871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:38.865871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:38.865871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:38.865871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:38.865871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:38.865871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:38.865871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:38.865871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:38.865871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:38.865871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:38.865871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:38.865871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:38.866222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:38.866222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:38.866222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:38.866222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:38.866222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:38.866222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:38.866222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:38.866222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:38.866222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:38.866222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:38.866222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:38.866222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:38.866222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:38.866222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:38.866222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:38.866222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:38.866222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:38.866222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:38.866222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:38.866580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:38.866580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:38.866580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:42.622633 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:42.623214 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:42.623214 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:42.623883 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:42.624091 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:42.624243 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:42.624413 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:42.624558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.625004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:42.625004 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:42.625004 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:42.625004 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:42.625004 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:42.625004 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:42.625004 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.625004 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:42.626033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.626033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:42.626033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:42.626033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:42.626033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.626033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:42.626033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:42.626033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.626033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:42.626033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:42.626033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:42.626033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.626406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:42.626406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:42.626406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:42.626406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:42.626406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:42.626406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:42.626406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:42.626406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:42.626406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:42.626406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:42.626406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:42.626406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:42.626406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:42.626840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.626840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:42.626840 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:42.627815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.627815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:42.627815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:42.627815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:42.627815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:42.627815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:42.627815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.627815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:42.627815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:42.627815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.627815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:42.627815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:42.627815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.627815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:42.627815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:42.627815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.628287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:42.628287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:42.628287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.628287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:42.628287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:42.628287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:42.628287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.628287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:42.628287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:42.628287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:42.628287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:42.628287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:42.628287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:42.628287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:42.628287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:42.628287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:42.628287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:42.628287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:42.628287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:42.628796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:42.628796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.628796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:42.628796 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:42.628796 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:42.628796 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:43:42.629165 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:42.629388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.629806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:42.629806 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:42.629806 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:42.629806 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:42.629806 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:42.629806 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:42.629806 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.629806 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:42.630660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.630660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:42.630660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:42.630660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:42.630660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.630660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:42.630660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:42.630660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:42.630660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.630660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:42.630660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:42.630660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:42.630660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:42.630660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:42.630660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:42.630942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:42.630942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:42.630942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:42.630942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:42.630942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:42.630942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:42.630942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:42.630942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.630942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:42.631830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.631830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:42.631830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:42.631830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:42.631830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.631830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:42.631830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:42.631830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:43:42.631830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:42.631830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:42.631830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.631830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:42.632706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:42.632706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:42.632706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.632706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:42.632706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:42.632706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:42.632706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:42.632706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:42.632706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:42.632706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:42.632706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:42.632706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:42.632706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:42.632706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:42.633161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:42.633161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:42.633161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.633161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:42.633161 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:42.633699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.633699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:42.633699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:42.633699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:42.633699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:42.633699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:42.633699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.633699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:42.633699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:42.633699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.633699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:42.633699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:42.633699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.633699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:42.633699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:42.633699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.634215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:42.634215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:42.634215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:43:42.634215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:42.634215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:42.634215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.634215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:42.634215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:42.634215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:42.634215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.634215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:42.634215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:42.634215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:42.634215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:42.634215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:42.634215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:42.634215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:42.634215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:42.634751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:42.634751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:42.634751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:42.634751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:42.634751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:42.634751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.634751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:42.634751 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:42.634751 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:42.634751 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:42.635181 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:42.635434 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:42.635693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.636172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:42.636172 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:42.636172 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:42.636172 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:42.636172 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:42.636172 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:42.636172 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.636172 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:42.636937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.636937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:42.636937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:42.636937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:42.636937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.636937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:42.636937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:42.636937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:42.636937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.636937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:42.636937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:42.636937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:42.636937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:42.636937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:42.636937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:42.637446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:42.637446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:42.637446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:42.637446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:42.637446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:42.637446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:42.637446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:42.637446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.637446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:42.638177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.638177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:42.638177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:42.638177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:42.638177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.638177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:42.638177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:42.638177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:42.638177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:42.638177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.638177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:42.638472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:42.638472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:42.638472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.638472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:42.638472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:42.638472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:42.638472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:42.638472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:42.638472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:42.638472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:42.638472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:42.638472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:42.638472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:42.638472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:42.638732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:42.638732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:42.638732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.638732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:42.638981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.638981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:42.638981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:42.638981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:42.638981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:42.638981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:42.638981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.638981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:42.638981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:42.638981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.638981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:42.638981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:42.638981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.638981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:42.638981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:42.638981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.638981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:42.639278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:42.639278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:42.639278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:42.639278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.639278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:42.639278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:42.639278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:42.639278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.639278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:42.639278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:42.639278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:42.639278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:42.639278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:42.639278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:42.639278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:42.639278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:42.639278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:42.639278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:42.639278 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:42.639595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:42.639595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:42.639595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.639595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:42.639595 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:42.639595 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:42.640221 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:42.640641 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:42.640902 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:42.641146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.641635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:42.641635 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:42.641635 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:42.641635 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:42.641635 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:42.641635 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:42.641635 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.641635 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:42.642511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.642511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:42.642511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:42.642511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:42.642511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.642511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:42.642511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:42.642511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.642511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:42.642511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:42.642511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:42.642511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.643138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:42.643138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:42.643138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:42.643138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:42.643138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:42.643138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:42.643138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:42.643138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:42.643138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:42.643138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:42.643138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:42.643138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:42.643138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:42.643386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.643386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:42.643808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.643808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:42.643808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:42.643808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:42.643808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:42.643808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:42.643808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.643808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:42.643808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:42.643808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.643808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:42.643808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:42.643808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.643808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:42.643808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:42.643808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:42.644098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:42.644098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:42.644098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.644098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:42.644098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:42.644098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:42.644098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:42.644098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:42.644098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:42.644098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:42.644098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:42.644098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:42.644098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:42.644098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:42.644098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:42.644098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:42.644098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:42.644098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:42.644098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:42.644432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:42.644432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:42.644432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:43.826678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:43.827205 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:43.827205 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:43.827730 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:43.863761 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:43.864057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:43.864612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:43.864612 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:43.864612 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:43.864612 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:43.864612 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:43.864612 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:43.864612 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:43.864612 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:43.865634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:43.865634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:43.865634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:43.865634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:43.865634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:43.865634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:43.865634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:43.865634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:43.865634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:43.865634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:43.865634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:43.865634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:43.866123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:43.866123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:43.866123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:43.866123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:43.866123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:43.866123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:43.866123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:43.866123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:43.866123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:43.866123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:43.866123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:43.866123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:43.866123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:43.866591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:43.866591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:43.867269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:43.867269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:43.867269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:43.867269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:43.867269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:43.867269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:43.867269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:43.867269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:43.867269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:43.867269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:43.867269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:43.867269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:43.867269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:43.867269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:43.867269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:43.867269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:43.867941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:43.867941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:43.867941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:43.867941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:43.867941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:43.867941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:43.867941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:43.867941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:43.867941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:43.867941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:43.867941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:43.867941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:43.867941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:43.867941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:43.867941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:43.867941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:43.867941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:43.867941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:43.867941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:43.868631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:43.868631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:43.868631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:47.623165 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:47.624972 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:47.624972 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:47.625403 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:47.625787 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:47.626051 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:47.626302 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:47.626632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.627182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:47.627182 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:47.627182 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:47.627182 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:47.627182 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:47.627182 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:47.627182 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.627182 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:47.628700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.628700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:47.628700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:47.628700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:47.628700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.628700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:47.628700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:47.628700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.628700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:47.628700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:47.628700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:47.628700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.630940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:47.630940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:47.630940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:47.630940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:47.630940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:47.630940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:47.630940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:47.630940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:47.630940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:47.630940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:47.630940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:47.630940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:47.630940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:47.631199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.631199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:47.631199 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:47.631199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.631199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:47.631199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:47.631199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:47.631199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:47.631199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:47.631199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.631199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:47.631199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:47.631199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.631489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:47.631489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:47.631489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.631489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:47.631489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:47.631489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.631489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:47.631489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:47.631489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.631489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:47.631489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:47.631489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:47.631489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.631489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:47.631489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:47.631489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:47.631489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:47.631489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:47.631489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:47.631881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:47.631881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:47.631881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:47.631881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:47.631881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:47.631881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:47.631881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:47.631881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.631881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:47.631881 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:47.631881 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:47.631881 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:43:47.633444 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:47.633444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.633815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:47.633815 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:47.633815 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:47.633815 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:47.633815 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:47.633815 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:47.633815 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.633815 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:47.637268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:47.637268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.637268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:47.637268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:47.637268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:47.637268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.637268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:47.637268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:47.637268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:47.637268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.637268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:47.637268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:47.637268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:47.637638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:47.637638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:47.637638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:47.637638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:47.637638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:47.637638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:47.637638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:47.637638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:47.637638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:47.637638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:47.637638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.637638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:47.637638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.637638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:47.638035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:47.638035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:47.638035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.638035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:47.638035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:47.638035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:43:47.638035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:47.638035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:47.638035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.638035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:47.638035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:47.638035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:47.638035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.638380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:47.638380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:47.638380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:47.638380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:47.638380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:47.638380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:47.638380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:47.638380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:47.638380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:47.638380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:47.638380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:47.638380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:47.638380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:47.638713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.638713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:47.638713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.638713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:47.638713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:47.638713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:47.638713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:47.638713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:47.638713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.638713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:47.638713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:47.638713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.638713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:47.638713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:47.638713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.639098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:47.639098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:47.639098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.639098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:47.639098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:47.639098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:43:47.639098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:47.639098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:47.639098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.639098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:47.639098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:47.639098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:47.639098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.639098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:47.639098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:47.639098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:47.639098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:47.639098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:47.639098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:47.639588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:47.639588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:47.639588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:47.639588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:47.639588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:47.639588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:47.639588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:47.639588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.639588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:47.639588 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:47.639588 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:47.641049 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:47.641360 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:47.641820 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:47.642125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.642649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:47.642649 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:47.642649 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:47.642649 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:47.642649 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:47.642649 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:47.642649 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.642649 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:47.645449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.645449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:47.645449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:47.645449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:47.645449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.645449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:47.645449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:47.645449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:47.645449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.645449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:47.645449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:47.645449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:47.645449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:47.645449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:47.645449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:47.645805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:47.645805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:47.645805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:47.645805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:47.645805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:47.645805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:47.645805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:47.645805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.645805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:47.645805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.645805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:47.645805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:47.645805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:47.645805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.646113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:47.646113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:47.646113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:47.646113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:47.646113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.646113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:47.646113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:47.646113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:47.646113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.646113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:47.646113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:47.646113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:47.646113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:47.646354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:47.646354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:47.646354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:47.646354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:47.646354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:47.646354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:47.646354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:47.646354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:47.646354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:47.646354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.646354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:47.646354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.646631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:47.646631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:47.646631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:47.646631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:47.646631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:47.646631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.646631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:47.646631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:47.646631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.646631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:47.646631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:47.646631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.646631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:47.646631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:47.646631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.646631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:47.646631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:47.646631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:47.646631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.646980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:47.647669 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:47.647768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:47.647949 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:47.648509 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:47.649369 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:47.649369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.649557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:47.649557 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:47.649557 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:47.649557 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:47.649557 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:47.649557 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:47.649557 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.649557 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:47.650130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.650130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:47.650130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:47.650130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:47.650130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.650130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:47.650130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:47.650130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.650130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:47.650130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:47.650130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:47.650130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.650394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:47.650394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:47.650394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:47.650394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:47.650394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:47.650394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:47.650394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:47.650394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:47.650394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:47.650394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:47.650394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:47.650394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:47.650394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:47.650602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.650602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:47.651016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.651016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:47.651016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:47.651016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:47.651016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:47.651016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:47.651016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.651016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:47.651016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:47.651016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.651016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:47.651016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:47.651016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.651016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:47.651016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:47.651016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:47.651327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:47.651327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:47.651327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.651327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:47.651327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:47.651327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:47.651327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:47.651327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:47.651327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:47.651327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:47.651327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:47.651327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:47.651327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:47.651327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:47.651327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:47.651327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:47.651327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:47.651327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:47.651327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:47.651796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:47.651796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:47.651796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:48.827117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:48.827608 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:48.827771 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:48.828297 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:48.857505 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:48.860267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:48.860267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:48.860267 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:48.860267 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:48.860267 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:48.860267 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:48.860267 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:48.860267 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:48.860267 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:48.860267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:48.860267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:48.860267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:48.860267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:48.860267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:48.860267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:48.860708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:48.860708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:48.860708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:48.860708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:48.860708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:48.860708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:48.860708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:48.860708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:48.860708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:48.860708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:48.860708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:48.860708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:48.860708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:48.860708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:48.860985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:48.860985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:48.860985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:48.860985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:48.860985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:48.860985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:48.860985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:48.860985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:48.860985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:48.860985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:48.860985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:48.860985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:48.860985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:48.861210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:48.861583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:48.861583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:48.861583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:48.861583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:48.861583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:48.861583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:48.861583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:48.861583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:48.861583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:48.861583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:48.861583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:48.861583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:52.624126 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:52.624647 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:52.624647 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:52.625087 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:52.625209 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:52.625403 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:52.625483 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:52.625592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.626028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:52.626028 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:52.626028 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:52.626028 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:52.626028 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:52.626028 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:52.626028 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.626028 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:52.626712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.626712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:52.626712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:52.626712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:52.626712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.626712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:52.626712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:52.626712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.626712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:52.626712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:52.626712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:52.626712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.626954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:52.626954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:52.626954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:52.626954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:52.626954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:52.626954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:52.626954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:52.626954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:52.626954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:52.626954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:52.626954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:52.626954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:52.626954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:52.627148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.627148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:52.627443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.627443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:52.627443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:52.627443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:52.627443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:52.627443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:52.627443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.627443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:52.627443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:52.627443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.627443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:52.627443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:52.627443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.627443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:52.627443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:52.627443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.627742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:52.627742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:52.627742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.627742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:52.627742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:52.627742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:52.627742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.627742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:52.627742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:52.627742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:52.627742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:52.627742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:52.627742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:52.627742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:52.627742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:52.627742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:52.627742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:52.627742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:52.627742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:52.628016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:52.628016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.628016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:52.630380 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:52.630585 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:52.630667 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:52.630762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:43:52.630984 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:52.631120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.631747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:52.631747 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:52.631747 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:52.631747 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:52.631747 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:52.631747 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:52.631747 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.631747 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:52.632269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.632269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:52.632269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:52.632269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:52.632269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.632269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:52.632269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:52.632269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:52.632269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.632269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:52.632269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:52.632269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:52.632269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:52.632269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:52.632269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:52.632583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:52.632583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:52.632583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:52.632583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:52.632583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:52.632583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:52.632583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:52.632583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.632583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:52.632957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.632957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:52.632957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:52.632957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:52.632957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.632957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:52.632957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:52.632957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:43:52.632957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:52.632957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:52.632957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.632957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:52.633596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:52.633596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:52.633596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.633596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:52.633596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:52.633596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:52.633596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:52.633596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:52.633596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:52.633596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:52.633596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:52.633596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:52.633596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:52.633596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:52.633904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:52.633904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:52.633904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.633904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:52.633970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.633970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:52.633970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:52.633970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:52.633970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:52.633970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:52.633970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.633970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:52.633970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:52.633970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.633970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:52.633970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:52.633970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.633970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:52.633970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:52.633970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.634217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:52.634217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:52.634217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:43:52.634217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:52.634217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:52.634217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.634217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:52.634217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:52.634217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:52.634217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.634217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:52.634217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:52.634217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:52.634217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:52.634217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:52.634217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:52.634217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:52.634217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:52.634571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:52.634571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:52.634571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:52.634571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:52.634571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:52.634571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.634571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:52.637069 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:52.637261 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:52.637353 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:52.637472 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:52.637709 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:52.637876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.638210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:52.638210 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:52.638210 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:52.638210 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:52.638210 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:52.638210 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:52.638210 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.638210 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:52.638850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.638850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:52.638850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:52.638850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:52.638850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.638850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:52.638850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:52.638850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:52.638850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.638850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:52.638850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:52.638850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:52.638850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:52.638850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:52.638850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:52.639105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:52.639105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:52.639105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:52.639105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:52.639105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:52.639105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:52.639105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:52.639105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.639105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:52.639503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.639503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:52.639503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:52.639503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:52.639503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.639503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:52.639503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:52.639503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:52.639503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:52.639503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.639503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:52.639741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:52.639741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:52.639741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.639741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:52.639741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:52.639741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:52.639741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:52.639741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:52.639741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:52.639741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:52.639741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:52.639741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:52.639741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:52.639741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:52.639971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:52.639971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:52.639971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.639971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:52.640328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.640328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:52.640328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:52.640328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:52.640328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:52.640328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:52.640328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.640328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:52.640328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:52.640328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.640328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:52.640328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:52.640328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.640328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:52.640328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:52.640328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.640328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:52.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:52.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:52.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:52.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:52.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:52.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:52.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:52.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:52.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:52.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:52.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:52.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:52.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:52.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:52.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:52.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:52.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:52.641303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:52.641303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:52.641303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.641303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:52.647791 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:52.648111 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:52.648267 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:52.648473 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:52.648760 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:52.648973 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:52.649106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.649502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:52.649502 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:52.649502 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:52.649502 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:52.649502 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:52.649502 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:52.649502 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.649502 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:52.650080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.650080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:52.650080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:52.650080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:52.650080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.650080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:52.650080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:52.650080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.650080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:52.650080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:52.650080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:52.650080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.650328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:52.650328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:52.650328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:52.650328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:52.650328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:52.650328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:52.650328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:52.650328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:52.650328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:52.650328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:52.650328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:52.650328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:52.650328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:52.650614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.650614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:52.650895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.650895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:52.650895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:52.650895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:52.650895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:52.650895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:52.650895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.650895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:52.650895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:52.650895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.650895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:52.650895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:52.650895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.650895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:52.650895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:52.650895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:52.651802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:52.651802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:52.651802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.651802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:52.651802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:52.651802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:52.651802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:52.651802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:52.651802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:52.651802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:52.651802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:52.651802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:52.651802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:52.651802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:52.651802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:52.651802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:52.651802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:52.651802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:52.651802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:52.652242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:52.652242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:52.652242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:53.827880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:53.828459 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:53.828678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:53.829527 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:53.830989 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=238 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:43:53.874378 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:53.874679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:53.875089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:53.875089 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:53.875089 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:53.875089 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:53.875089 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:53.875089 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:53.875089 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:53.875089 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:53.875857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:53.875857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:53.875857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:53.875857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:53.875857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:53.875857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:53.875857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:53.875857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:53.875857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:53.875857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:53.875857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:53.875857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:53.876134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:53.876134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:53.876134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:53.876134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:53.876134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:53.876134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:53.876134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:53.876134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:53.876134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:53.876134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:53.876134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:53.876134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:53.876134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:53.876451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:53.876451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:53.876913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:53.876913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:53.876913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:53.876913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:53.876913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:53.876913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:53.876913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:53.876913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:53.876913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:53.876913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:53.876913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:53.876913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:53.876913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:53.876913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:53.876913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:53.876913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:53.877754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:53.877754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:53.877754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:53.877754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:53.877754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:53.877754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:53.877754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:53.877754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:53.877754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:53.877754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:53.877754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:53.877754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:53.877754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:53.877754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:53.877754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:53.877754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:53.877754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:53.877754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:53.877754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:53.878138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:53.878138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:53.878138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:57.624520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:57.624901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:57.624991 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:57.626082 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:57.626310 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:57.626476 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:57.626630 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:57.626741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.627211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:57.627211 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:57.627211 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:57.627211 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:57.627211 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:57.627211 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:57.627211 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.627211 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:57.627983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.627983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:57.627983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:57.627983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:57.627983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.627983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:57.627983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:57.627983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.627983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:57.627983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:57.627983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:57.627983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.628276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:57.628276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:57.628276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:57.628276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:57.628276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:57.628276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:57.628276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:57.628276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:57.628276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:57.628276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:57.628276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:57.628276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:57.628276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:57.628510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.628510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:57.628820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.628820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:57.628820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:57.628820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:57.628820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:57.628820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:57.628820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.628820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:57.628820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:57.628820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.628820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:57.628820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:57.628820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.628820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:57.628820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:57.628820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.629100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:43:57.629100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:57.629100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.629100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:57.629100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:57.629100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:57.629100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.629100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:57.629100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:57.629100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:57.629100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:57.629100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:57.629100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:57.629100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:57.629100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:57.629100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:57.629100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:57.629100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:57.629100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:57.629437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:57.629437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.629437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:57.630649 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:57.630829 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:57.630907 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:57.631010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:43:57.631438 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:57.631640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.632154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:57.632154 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:57.632154 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:57.632154 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:57.632154 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:57.632154 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:57.632154 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.632154 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:57.632800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.632800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:57.632800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:57.632800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:57.632800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.632800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:57.632800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:57.632800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:57.632800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.632800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:57.632800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:57.632800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:57.632800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:57.632800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:57.632800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:57.633310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:57.633310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:57.633310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:57.633310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:57.633310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:57.633310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:57.633310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:57.633310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.633310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:57.633567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.633567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:57.633567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:57.633567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:57.633567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.633567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:57.633567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:57.633567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:43:57.633567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:57.633567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:57.633567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.633567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:57.633787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:57.633787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:57.633787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.633787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:57.633787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:57.633787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:57.633787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:57.633787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:57.633787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:57.633787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:57.633787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:57.633787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:57.633787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:57.633787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:57.634017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:57.634017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:57.634017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.634017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:57.634518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.634518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:57.634518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:57.634518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:57.634518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:57.634518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:57.634518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.634518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:57.634518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:57.634518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.634518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:57.634518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:57.634518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.634518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:57.634518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:57.634518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.634824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:43:57.634824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:43:57.634824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:43:57.634824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:57.634824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:57.634824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.634824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:57.634824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:57.634824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:57.634824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.634824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:57.634824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:57.634824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:57.634824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:57.634824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:57.634824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:57.634824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:57.634824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:57.635133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:57.635133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:57.635133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:57.635133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:57.635133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:57.635133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.635133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:57.637363 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:57.637604 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:57.637736 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:57.637859 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:57.638262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:57.638509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.639028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:57.639028 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:57.639028 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:57.639028 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:57.639028 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:57.639028 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:57.639028 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.639028 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:57.639661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.639661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:43:57.639661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:57.639661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:57.639661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.639661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:57.639661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:57.639661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:43:57.639661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.639661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:43:57.639661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:57.639661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:43:57.639661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:57.639661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:43:57.639661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:43:57.639916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:57.639916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:43:57.639916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:57.639916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:43:57.639916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:43:57.639916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:57.639916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:43:57.639916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.639916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:43:57.640384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.640384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:57.640384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:57.640384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:57.640384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.640384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:57.640384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:57.640384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:57.640384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:57.640384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.640384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:57.640614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:57.640614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:57.640614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.640614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:57.640614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:57.640614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:57.640614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:57.640614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:57.640614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:57.640614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:57.640614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:57.640614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:57.640614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:57.640614 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:57.640849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:57.640849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:57.640849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.640849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:57.641561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.641561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:57.641561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:57.641561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:57.641561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:57.641561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:57.641561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.641561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:57.641561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:57.641561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.641561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:57.641561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:57.641561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.641561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:57.641561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:57.641561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.641561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:43:57.641879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:43:57.641879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:43:57.641879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:43:57.641879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.641879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:57.641879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:57.641879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:57.641879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.641879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:57.641879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:57.641879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:57.641879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:57.641879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:57.641879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:57.641879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:57.641879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:57.641879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:57.641879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:57.641879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:57.642535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:57.642535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:57.642535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.642535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:57.648331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:57.648596 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:57.648673 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:57.648825 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:57.649212 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:57.649461 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:57.649668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.650177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:57.650177 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:57.650177 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:57.650177 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:57.650177 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:57.650177 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:57.650177 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.650177 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:57.650836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.650836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:57.650836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:57.650836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:57.650836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.650836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:57.650836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:57.650836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.650836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:57.650836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:57.650836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:57.650836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.651090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:57.651090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:57.651090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:57.651090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:57.651090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:57.651090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:57.651090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:57.651090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:57.651090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:57.651090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:57.651090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:57.651090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:57.651090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:57.651318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.651318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:57.651544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.651544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:57.651544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:57.651544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:57.651544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:57.651544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:57.651544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.651544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:57.651544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:57.651544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.651544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:57.651544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:57.651544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.651544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:57.651544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:57.651544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:57.651848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:43:57.651848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:57.651848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.651848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:57.651848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:57.651848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:57.651848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:57.651848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:57.651848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:57.651848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:57.651848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:57.651848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:57.651848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:57.651848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:57.651848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:57.651848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:57.651848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:57.651848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:57.651848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:57.652163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:57.652163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:57.652163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:43:58.828389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:43:58.829520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:43:58.829659 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:43:58.830292 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:43:58.860902 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:43:58.861143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:58.861671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:43:58.861671 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:58.861671 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:43:58.861671 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:43:58.861671 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:58.861671 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:43:58.861671 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:58.861671 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:43:58.862702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:58.862702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:43:58.862702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:58.862702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:43:58.862702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:58.862702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:58.862702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:58.862702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:58.862702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:58.862702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:58.862702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:43:58.862702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:58.862941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:43:58.862941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:58.862941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:43:58.862941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:58.862941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:43:58.862941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:43:58.862941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:58.862941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:43:58.862941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:58.862941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:43:58.862941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:43:58.862941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:58.862941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:43:58.863135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:58.863135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:43:58.863745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:58.863745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:43:58.863745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:43:58.863745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:43:58.863745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:43:58.863745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:43:58.863745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:58.863745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:43:58.863745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:43:58.863745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:58.863745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:43:58.863745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:43:58.863745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:43:58.863745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:43:58.863745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:43:58.863745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:43:58.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:43:58.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:43:58.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:58.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:58.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:43:58.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:43:58.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:43:58.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:43:58.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:43:58.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:43:58.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:43:58.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:43:58.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:43:58.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:43:58.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:43:58.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:43:58.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:43:58.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:43:58.864199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:43:58.864668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:43:58.864668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:43:58.864668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:02.625041 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:02.626288 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:02.626288 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:02.626586 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:02.626772 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:02.626936 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:02.627101 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:02.627272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.629296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:02.629296 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:02.629296 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:02.629296 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:02.629296 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:02.629296 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:02.629296 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.629296 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:02.629296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.629296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:02.629296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:02.629296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:02.629296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.629296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:02.629296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:02.629653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.629653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:02.629653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:02.629653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:02.629653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.629653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:02.629653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:02.629653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:02.629653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:02.629653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:02.629653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:02.629653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:02.629653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:02.629653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:02.629864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:02.629864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:02.629864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:02.629864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:02.629864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.629864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:02.629864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.629864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:02.629864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:02.629864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:02.629864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:02.629864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:02.629864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.629864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:02.630103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:02.630441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:02.630441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:02.630441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:02.630441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:02.630441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:02.630441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:02.630441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:02.630441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:02.630441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.630441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:02.631096 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:02.631449 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:02.631706 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:02.631926 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:44:02.632423 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:02.632654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.633620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:02.633620 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:02.633620 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:02.633620 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:02.633620 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:02.633620 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:02.633620 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.633620 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:02.634295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.634295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:02.634295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:02.634295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:02.634295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.634295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:02.634295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:02.634295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:02.634295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.634295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:02.634295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:02.634295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:02.634295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:02.634295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:02.634295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:02.634653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:02.634653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:02.634653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:02.634653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:02.634653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:02.634653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:02.634653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:02.634653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.634653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:02.635510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.635510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:02.635510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:02.635510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:02.635510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.635510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:02.635510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:02.635510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:44:02.635510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:02.635510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:02.635510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.635510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:02.635788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:02.635788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:02.635788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.635788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:02.635788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:02.635788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:02.635788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:02.635788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:02.635788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:02.635788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:02.635788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:02.635788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:02.635788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:02.635788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:02.636012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:02.636012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:02.636012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.636012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:02.636397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.636397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:02.636397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:02.636397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:02.636397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:02.636397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:02.636397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.636397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:02.636397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:02.636397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.636397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:02.636397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:02.636397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.636397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:02.636397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:02.636397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.636749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:02.636749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:02.636749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:44:02.636749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:02.636749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:02.636749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.636749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:02.636749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:02.636749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:02.636749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.636749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:02.636749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:02.636749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:02.636749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:02.636749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:02.636749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:02.636749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:02.636749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:02.637023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:02.637023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:02.637023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:02.637023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:02.637023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:02.637023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.637023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:02.637796 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:02.638080 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:02.638203 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:02.638345 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:02.638688 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:02.638884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.639296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:02.639296 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:02.639296 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:02.639296 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:02.639296 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:02.639296 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:02.639296 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.639296 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:02.639861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.639861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:02.639861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:02.639861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:02.639861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.639861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:02.639861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:02.639861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:02.639861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.639861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:02.639861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:02.639861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:02.639861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:02.639861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:02.639861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:02.640177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:02.640177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:02.640177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:02.640177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:02.640177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:02.640177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:02.640177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:02.640177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.640177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:02.640576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.640576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:02.640576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:02.640576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:02.640576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.640576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:02.640576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:02.640576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:02.640576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:02.640576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.640576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:02.640813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:02.640813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:02.640813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.640813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:02.640813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:02.640813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:02.640813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:02.640813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:02.640813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:02.640813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:02.640813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:02.640813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:02.640813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:02.640813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:02.641043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:02.641043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:02.641043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.641043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:02.641428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.641428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:02.641428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:02.641428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:02.641428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:02.641428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:02.641428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.641428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:02.641428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:02.641428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.641428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:02.641428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:02.641428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.641428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:02.641428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:02.641428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.641428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:02.641824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:02.641824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:02.641824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:02.641824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.641824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:02.641824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:02.641824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:02.641824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.641824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:02.641824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:02.641824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:02.641824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:02.641824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:02.641824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:02.641824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:02.641824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:02.641824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:02.641824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:02.641824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:02.642119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:02.642119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:02.642119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.642119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:02.648818 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:02.649196 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:02.649353 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:02.649585 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:02.649969 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:02.651179 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:02.651179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.651338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:02.651338 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:02.651338 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:02.651338 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:02.651338 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:02.651338 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:02.651338 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.651338 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:02.651992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.651992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:02.651992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:02.651992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:02.651992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.651992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:02.651992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:02.651992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.651992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:02.651992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:02.651992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:02.651992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.652455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:02.652455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:02.652455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:02.652455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:02.652455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:02.652455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:02.652455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:02.652455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:02.652455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:02.652455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:02.652455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:02.652455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:02.652455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:02.652952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.652952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:02.652952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.652952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:02.652952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:02.652952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:02.652952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:02.652952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:02.652952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.652952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:02.652952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:02.652952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.652952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:02.652952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:02.652952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.653458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:02.653458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:02.653458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:02.653458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:02.653458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:02.653458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.653458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:02.653458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:02.653458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:02.653458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:02.653458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:02.653458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:02.653458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:02.653458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:02.653458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:02.653458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:02.653458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:02.653458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:02.653458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:02.654070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:02.654070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:02.654070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:02.654070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:02.654070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:02.654070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:03.829711 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:03.830228 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:03.830336 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:03.831135 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:03.873668 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:03.874161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:03.874916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:03.874916 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:03.874916 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:03.874916 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:03.874916 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:03.874916 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:03.874916 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:03.874916 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:03.875861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:03.875861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:03.875861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:03.875861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:03.875861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:03.875861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:03.875861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:03.875861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:03.875861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:03.875861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:03.875861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:03.875861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:03.876333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:03.876333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:03.876333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:03.876333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:03.876333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:03.876333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:03.876333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:03.876333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:03.876333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:03.876333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:03.876333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:03.876333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:03.876333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:03.876799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:03.876799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:03.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:03.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:03.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:03.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:03.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:03.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:03.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:03.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:03.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:03.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:03.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:03.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:03.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:03.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:03.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:03.877197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:03.877775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:03.877775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:03.877775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:03.877775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:03.877775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:03.877775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:03.877775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:03.877775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:03.877775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:03.877775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:03.877775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:03.877775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:03.877775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:03.877775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:03.877775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:03.877775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:03.877775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:03.877775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:03.877775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:03.878475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:03.878475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:03.878475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:07.626687 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:07.627416 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:07.627416 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:07.628503 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:07.628738 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:07.628941 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:07.629002 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:07.629151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.629712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:07.629712 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:07.629712 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:07.629712 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:07.629712 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:07.629712 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:07.629712 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.629712 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:07.630475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.630475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:07.630475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:07.630475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:07.630475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.630475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:07.630475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:07.630475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.630475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:07.630475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:07.630475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:07.630475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.630974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:07.630974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:07.630974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:07.630974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:07.630974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:07.630974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:07.630974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:07.630974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:07.630974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:07.630974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:07.630974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:07.630974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:07.630974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:07.631324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.631324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:07.631324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.631324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:07.631324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:07.631324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:07.631324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:07.631324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:07.631324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.631324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:07.631324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:07.631324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.631324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:07.631324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:07.631324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.631801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:07.631801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:07.631801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.631801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:07.631801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:07.631801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.631801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:07.631801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:07.631801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:07.631801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.631801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:07.631801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:07.631801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:07.631801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:07.631801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:07.631801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:07.631801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:07.631801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:07.631801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:07.632284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:07.632284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:07.632284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:07.632284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:07.632284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.632284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:07.632284 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:07.632284 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:07.632284 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:07.632284 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:44:07.632619 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:07.632619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.632845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:07.632845 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:07.632845 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:07.632845 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:07.632845 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:07.632845 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:07.632845 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.632845 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:07.633679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.633679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:07.633679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:07.633679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:07.633679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.633679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:07.633679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:07.633679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:07.633679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.633679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:07.633679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:07.633679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:07.633679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:07.633679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:07.633679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:07.634293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:07.634293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:07.634293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:07.634293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:07.634293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:07.634293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:07.634293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:07.634293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.634293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:07.634627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.634627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:07.634627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:07.634627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:07.634627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.634627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:07.634627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:07.634627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:44:07.634627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:07.634627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:07.634627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.634627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:07.634954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:07.634954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:07.634954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.634954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:07.634954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:07.634954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:07.634954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:07.634954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:07.634954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:07.634954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:07.634954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:07.634954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:07.634954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:07.634954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:07.635434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:07.635434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:07.635434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.635434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:07.635739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.635739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:07.635739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:07.635739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:07.635739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:07.635739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:07.635739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.635739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:07.635739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:07.635739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.635739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:07.635739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:07.635739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.635739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:07.635739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:07.635739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.636348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:07.636348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:07.636348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:44:07.636348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:07.636348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:07.636348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.636348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:07.636348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:07.636348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:07.636348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.636348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:07.636348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:07.636348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:07.636348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:07.636348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:07.636348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:07.636348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:07.636348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:07.636820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:07.636820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:07.636820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:07.636820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:07.636820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:07.636820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.636820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:07.638197 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:07.638491 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:07.638650 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:07.638803 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:07.639210 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:07.639463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.640081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:07.640081 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:07.640081 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:07.640081 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:07.640081 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:07.640081 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:07.640081 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.640081 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:07.640831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.640831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:07.640831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:07.640831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:07.640831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.640831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:07.640831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:07.640831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:07.640831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.640831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:07.640831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:07.640831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:07.640831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:07.640831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:07.640831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:07.641321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:07.641321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:07.641321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:07.641321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:07.641321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:07.641321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:07.641321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:07.641321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.641321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:07.641843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.641843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:07.641843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:07.641843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:07.641843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.641843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:07.641843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:07.641843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:07.641843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:07.641843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.641843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:07.642330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:07.642330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:07.642330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.642330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:07.642330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:07.642330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:07.642330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:07.642330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:07.642330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:07.642330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:07.642330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:07.642330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:07.642330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:07.642330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:07.642749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:07.642749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:07.642749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.642749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:07.643010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.643010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:07.643010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:07.643010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:07.643010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:07.643010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:07.643010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.643010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:07.643010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:07.643010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.643010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:07.643010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:07.643010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.643010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:07.643010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:07.643010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.643010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:07.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:07.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:07.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:07.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:07.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:07.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:07.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:07.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:07.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:07.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:07.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:07.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:07.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:07.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:07.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:07.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:07.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:07.643782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:07.643782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:07.643782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.643782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:07.649449 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:07.650231 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:07.650231 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:07.650504 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:07.650797 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:07.651092 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:07.651346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.652101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:07.652101 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:07.652101 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:07.652101 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:07.652101 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:07.652101 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:07.652101 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.652101 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:07.653161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.653161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:07.653161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:07.653161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:07.653161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.653161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:07.653161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:07.653161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.653161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:07.653161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:07.653161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:07.653161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.653628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:07.653628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:07.653628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:07.653628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:07.653628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:07.653628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:07.653628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:07.653628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:07.653628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:07.653628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:07.653628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:07.653628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:07.653628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:07.653941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.653941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:07.654413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.654413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:07.654413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:07.654413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:07.654413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:07.654413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:07.654413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.654413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:07.654413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:07.654413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.654413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:07.654413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:07.654413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.654413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:07.654413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:07.654413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:07.654895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:07.654895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:07.654895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.654895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:07.654895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:07.654895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:07.654895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:07.654895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:07.654895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:07.654895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:07.654895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:07.654895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:07.654895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:07.654895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:07.654895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:07.654895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:07.654895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:07.654895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:07.654895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:07.655717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:07.655717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:07.655717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:08.830208 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:08.830746 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:08.830782 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:08.831610 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:08.879658 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:08.879924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:08.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:08.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:08.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:08.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:08.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:08.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:08.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:08.880599 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:08.881519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.050 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:08.881519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:08.881519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:08.881519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:08.881519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:08.881519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:08.881519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:08.881519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:08.881519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:08.881519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:08.881519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:08.881519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:08.881885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:08.881885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:08.881885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:08.881885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:08.881885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:08.881885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:08.881885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:08.881885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:08.881885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:08.881885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:08.881885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:08.881885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:08.881885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:08.882303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:08.882303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:08.882636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:08.882636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:08.882636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:08.882636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:08.882636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:08.882636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:08.882636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:08.882636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:08.882636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:08.882636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:08.882636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:08.882636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:08.882636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:08.882636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:08.882636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:08.882636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:08.883124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:08.883124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:08.883124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:08.883124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:08.883124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:08.883124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:08.883124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:08.883124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:08.883124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:08.883124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:08.883124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:08.883124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:08.883124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:08.883124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:08.883124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:08.883124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:08.883124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:08.883124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:08.883124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:08.883766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:08.883766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:08.883766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:12.627219 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:12.627750 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:12.627750 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:12.628724 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:12.629002 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:12.629184 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:12.629380 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:12.629656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.630212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:12.630212 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:12.630212 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:12.630212 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:12.630212 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:12.630212 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:12.630212 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.630212 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:12.631445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.631445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:12.631445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:12.631445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:12.631445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.631445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:12.631445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:12.631445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.631445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:12.631445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:12.631445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:12.631445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.631802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:12.631802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:12.631802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:12.631802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:12.631802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:12.631802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:12.631802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:12.631802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:12.631802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:12.631802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:12.631802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:12.631802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:12.631802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:12.632071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.632071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:12.632896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.632896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:12.632896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:12.632896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:12.632896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:12.632896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:12.632896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.632896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:12.632896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:12.632896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.632896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:12.632896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:12.632896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.632896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:12.632896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:12.632896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.633412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:12.633412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:12.633412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.633412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:12.633412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:12.633412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:12.633412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.633412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:12.633412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:12.633412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:12.633412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:12.633412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:12.633412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:12.633412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:12.633412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:12.633412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:12.633412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:12.633412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:12.633412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:12.634028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:12.634028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.634028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:12.634174 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:12.634582 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:12.634582 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:12.634684 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:44:12.635123 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:12.635307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:12.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:12.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:12.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:12.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:12.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:12.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.635785 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:12.636733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.636733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:12.636733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:12.636733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:12.636733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.636733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:12.636733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:12.636733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:12.636733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.636733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:12.636733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:12.636733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:12.636733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:12.636733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:12.636733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:12.637284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:12.637284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:12.637284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:12.637284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:12.637284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:12.637284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:12.637284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:12.637284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.637284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:12.637721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.637721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:12.637721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:12.637721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:12.637721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.637721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:12.637721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:12.637721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:44:12.637721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:12.637721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:12.637721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.637721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:12.638368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:12.638368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:12.638368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.638368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:12.638368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:12.638368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:12.638368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:12.638368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:12.638368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:12.638368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:12.638368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:12.638368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:12.638368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:12.638368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:12.639031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:12.639031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:12.639031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.639031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:12.639031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.639031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:12.639031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:12.639031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:12.639031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:12.639031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:12.639031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.639031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:12.639031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:12.639031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.639970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:12.639970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:12.639970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.639970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:12.639970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:12.639970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.639970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:12.639970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:12.639970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:44:12.639970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:12.639970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:12.639970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.639970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:12.639970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:12.639970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:12.639970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.639970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:12.639970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:12.639970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:12.640613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:12.640613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:12.640613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:12.640613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:12.640613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:12.640613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:12.640613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:12.640613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:12.640613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:12.640613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:12.640613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.640613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:12.640613 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:12.640613 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:12.641059 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:12.641059 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:12.641059 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:12.641059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.641059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:12.641059 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:12.641059 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:12.641059 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:12.641059 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:12.641059 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:12.641059 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.641059 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:12.641759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.641759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:12.641759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:12.641759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:12.641759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.641759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:12.641759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:12.641759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:12.641759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.641759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:12.641759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:12.641759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:12.641759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:12.641759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:12.641759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:12.642137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:12.642137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:12.642137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:12.642137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:12.642137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:12.642137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:12.642137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:12.642137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.642137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:12.642788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.642788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:12.642788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:12.642788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:12.642788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.642788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:12.642788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:12.642788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:12.642788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:12.642788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.642788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:12.643166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:12.643166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:12.643166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.643166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:12.643166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:12.643166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:12.643166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:12.643166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:12.643166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:12.643166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:12.643166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:12.643166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:12.643166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:12.643166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:12.643594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:12.643594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:12.643594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.643594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:12.644390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.644390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:12.644390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:12.644390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:12.644390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:12.644390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:12.644390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.644390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:12.644390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:12.644390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.644390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:12.644390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:12.644390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.644390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:12.644390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:12.644390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.644390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:12.644958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:12.644958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:12.644958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:12.644958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.644958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:12.644958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:12.644958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:12.644958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.644958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:12.644958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:12.644958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:12.644958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:12.644958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:12.644958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:12.644958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:12.644958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:12.644958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:12.644958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:12.644958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:12.645561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:12.645561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:12.645561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.645561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:12.649881 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:12.650153 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:12.650314 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:12.650513 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:12.650947 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:12.651195 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:12.651406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.651961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:12.651961 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:12.651961 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:12.651961 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:12.651961 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:12.651961 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:12.651961 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.651961 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:12.653020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.653020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:12.653020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:12.653020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:12.653020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.653020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:12.653020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:12.653020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.653020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:12.653020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:12.653020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:12.653020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.653482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:12.653482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:12.653482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:12.653482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:12.653482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:12.653482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:12.653482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:12.653482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:12.653482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:12.653482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:12.653482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:12.653482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:12.653482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:12.653872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.653872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:12.654526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.654526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:12.654526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:12.654526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:12.654526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:12.654526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:12.654526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.654526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:12.654526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:12.654526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.654526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:12.654526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:12.654526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.654526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:12.654526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:12.654526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:12.654932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:12.654932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:12.654932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.654932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:12.654932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:12.654932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:12.654932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:12.654932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:12.654932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:12.654932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:12.654932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:12.654932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:12.654932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:12.654932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:12.654932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:12.654932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:12.654932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:12.654932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:12.654932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:12.655592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:12.655592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:12.655592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:13.830884 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:13.831411 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:13.831411 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:13.831868 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:13.863099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:13.863372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:13.863731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:13.863731 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:13.863731 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:13.863731 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:13.863731 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:13.863731 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:13.863731 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:13.863731 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:13.864401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:13.864401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:13.864401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:13.864401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:13.864401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:13.864401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:13.864401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:13.864401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:13.864401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:13.864401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:13.864401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:13.864401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:13.864837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:13.864837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:13.864837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:13.864837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:13.864837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:13.864837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:13.864837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:13.864837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:13.864837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:13.864837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:13.864837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:13.864837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:13.864837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:13.865209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:13.865209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:13.865209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:13.865209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:13.865209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:13.865209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:13.865209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:13.865209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:13.865209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:13.865209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:13.865209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:13.865209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:13.865209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:13.865209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:13.865209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:13.865689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:13.865689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:13.865689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:13.865689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:13.865689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:13.865689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:13.865689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:13.865689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:13.865689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:13.865689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:13.865689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:13.865689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:13.865689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:13.865689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:13.865689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:13.865689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:13.865689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:13.865689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:13.865689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:13.866218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:13.866218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:13.866218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:13.866218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:13.866218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:13.866218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:17.627745 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:17.628325 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:17.628325 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:17.628898 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:17.629049 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:17.629158 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:17.629278 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:17.629399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.629850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:17.629850 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:17.629850 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:17.629850 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:17.629850 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:17.629850 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:17.629850 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.629850 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:17.630470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.630470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:17.630470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:17.630470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:17.630470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.630470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:17.630470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:17.630470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.630470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:17.630470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:17.630470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:17.630470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.630878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:17.630878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:17.630878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:17.630878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:17.630878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:17.630878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:17.630878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:17.630878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:17.630878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:17.630878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:17.630878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:17.630878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:17.630878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:17.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:17.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:17.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:17.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:17.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:17.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:17.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:17.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:17.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:17.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:17.631241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.631720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:17.631720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:17.631720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.631720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:17.631720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:17.631720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.631720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:17.631720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:17.631720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:17.631720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.631720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:17.631720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:17.631720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:17.631720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:17.631720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:17.631720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:17.631720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:17.631720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:17.631720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:17.632272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:17.632272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:17.632272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:17.632272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:17.632272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.632272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:17.634512 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:17.634787 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:17.634874 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:17.634981 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:44:17.635241 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:17.635395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.635778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:17.635778 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:17.635778 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:17.635778 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:17.635778 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:17.635778 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:17.635778 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.635778 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:17.636416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.636416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:17.636416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:17.636416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:17.636416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.636416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:17.636416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:17.636416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:17.636416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.636416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:17.636416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:17.636416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:17.636416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:17.636416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:17.636416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:17.636757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:17.636757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:17.636757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:17.636757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:17.636757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:17.636757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:17.636757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:17.636757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.636757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:17.636992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.636992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:17.636992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:17.636992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:17.636992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.636992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:17.636992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:17.636992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:44:17.636992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:17.636992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:17.636992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.636992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:17.637476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:17.637476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:17.637476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.637476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:17.637476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:17.637476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:17.637476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:17.637476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:17.637476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:17.637476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:17.637476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:17.637476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:17.637476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:17.637476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:17.638052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:17.638052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:17.638052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.638052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:17.638052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.638052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:17.638052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:17.638052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:17.638052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:17.638052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:17.638052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.638052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:17.638052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:17.638052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.638510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:17.638510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:17.638510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.638510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:17.638510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:17.638510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.638510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:17.638510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:17.638510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:44:17.638510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:17.638510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:17.638510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.638510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:17.638510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:17.638510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:17.638510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.638510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:17.638510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:17.638510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:17.639078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:17.639078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:17.639078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:17.639078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:17.639078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:17.639078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:17.639078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:17.639078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:17.639078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:17.639078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:17.639078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.639078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:17.639675 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:17.640028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:17.640128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:17.640230 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:17.640611 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:17.640820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.641215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:17.641215 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:17.641215 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:17.641215 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:17.641215 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:17.641215 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:17.641215 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.641215 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:17.641830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.641830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:17.641830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:17.641830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:17.641830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.641830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:17.641830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:17.641830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:17.641830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.641830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:17.641830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:17.641830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:17.641830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:17.641830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:17.641830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:17.642103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:17.642103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:17.642103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:17.642103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:17.642103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:17.642103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:17.642103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:17.642103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.642103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:17.642452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.642452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:17.642452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:17.642452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:17.642452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.642452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:17.642452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:17.642452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:17.642452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:17.642452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.642452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:17.642689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:17.642689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:17.642689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.642689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:17.642689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:17.642689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:17.642689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:17.642689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:17.642689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:17.642689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:17.642689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:17.642689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:17.642689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:17.642689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:17.642935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:17.642935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:17.642935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.642935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:17.643435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.643435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:17.643435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:17.643435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:17.643435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:17.643435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:17.643435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.643435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:17.643435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:17.643435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.643435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:17.643435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:17.643435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.643435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:17.643435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:17.643435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.643435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:17.643960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:17.643960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:17.643960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:17.643960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.643960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:17.643960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:17.643960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:17.643960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.643960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:17.643960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:17.643960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:17.643960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:17.643960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:17.643960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:17.643960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:17.643960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:17.643960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:17.643960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:17.643960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:17.644488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:17.644488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:17.644488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.644488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:17.650288 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:17.650748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:17.650866 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:17.651082 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:17.651471 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:17.651708 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:17.651856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.652371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:17.652371 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:17.652371 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:17.652371 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:17.652371 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:17.652371 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:17.652371 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.652371 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:17.653448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.653448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:17.653448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:17.653448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:17.653448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.653448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:17.653448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:17.653448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.653448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:17.653448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:17.653448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:17.653448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.653908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:17.653908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:17.653908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:17.653908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:17.653908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:17.653908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:17.653908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:17.653908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:17.653908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:17.653908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:17.653908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:17.653908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:17.653908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:17.654331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.654331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:17.654629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.654629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:17.654629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:17.654629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:17.654629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:17.654629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:17.654629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.654629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:17.654629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:17.654629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.654629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:17.654629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:17.654629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.654629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:17.654629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:17.654629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:17.655448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:17.655448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:17.655448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.655448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:17.655448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:17.655448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:17.655448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:17.655448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:17.655448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:17.655448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:17.655448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:17.655448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:17.655448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:17.655448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:17.655448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:17.655448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:17.655448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:17.655448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:17.655448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:17.656070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:17.656070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:17.656070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:18.831270 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:18.831794 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:18.831794 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:18.832227 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:18.876052 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:18.876311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:18.876871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:18.876871 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:18.876871 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:18.876871 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:18.876871 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:18.876871 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:18.876871 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:18.876871 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:18.877865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:18.877865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:18.877865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:18.877865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:18.877865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:18.877865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:18.877865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:18.877865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:18.877865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:18.877865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:18.877865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:18.877865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:18.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:18.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:18.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:18.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:18.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:18.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:18.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:18.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:18.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:18.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:18.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:18.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:18.878306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:18.878716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:18.878716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:18.879435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:18.879435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:18.879435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:18.879435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:18.879435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:18.879435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:18.879435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:18.879435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:18.879435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:18.879435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:18.879435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:18.879435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:18.879435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:18.879435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:18.879435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:18.879435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:18.879904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:18.879904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:18.879904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:18.879904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:18.879904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:18.879904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:18.879904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:18.879904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:18.879904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:18.879904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:18.879904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:18.879904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:18.879904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:18.879904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:18.879904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:18.879904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:18.879904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:18.879904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:18.879904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:18.880284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:18.880284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:18.880284 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:22.628210 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:22.628687 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:22.628687 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:22.629545 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:22.629747 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:22.629856 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:22.629968 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:22.630068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.630563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:22.630563 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:22.630563 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:22.630563 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:22.630563 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:22.630563 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:22.630563 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.630563 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:22.631287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.631287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:22.631287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:22.631287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:22.631287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.631287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:22.631287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:22.631287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.631287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:22.631287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:22.631287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:22.631287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.631662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:22.631662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:22.631662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:22.631662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:22.631662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:22.631662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:22.631662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:22.631662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:22.631662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:22.631662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:22.631662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:22.631662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:22.631662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:22.631965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.631965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:22.632021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.632021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:22.632021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:22.632021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:22.632021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:22.632021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:22.632021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.632021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:22.632021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:22.632021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.632021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:22.632021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:22.632021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.632021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:22.632021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:22.632021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.632643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:22.632643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:22.632643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.632643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:22.632643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:22.632643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:22.632643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.632643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:22.632643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:22.632643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:22.632643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:22.632643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:22.632643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:22.632643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:22.632643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:22.632643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:22.632643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:22.632643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:22.632643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:22.633158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:22.633158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.633158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:22.634917 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:22.635194 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:22.635428 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:22.635573 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:44:22.635877 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:22.636013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.636406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:22.636406 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:22.636406 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:22.636406 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:22.636406 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:22.636406 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:22.636406 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.636406 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:22.637048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.637048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:22.637048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:22.637048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:22.637048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.637048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:22.637048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:22.637048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:22.637048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.637048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:22.637048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:22.637048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:22.637048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:22.637048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:22.637048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:22.637445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:22.637445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:22.637445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:22.637445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:22.637445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:22.637445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:22.637445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:22.637445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.637445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:22.637675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.637675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:22.637675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:22.637675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:22.637675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.637675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:22.637675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:22.637675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:44:22.637675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:22.637675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:22.637675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.637675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:22.637967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:22.637967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:22.637967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.637967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:22.637967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:22.637967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:22.637967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:22.637967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:22.637967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:22.637967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:22.637967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:22.637967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:22.637967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:22.637967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:22.638396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:22.638396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:22.638396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.638396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:22.638396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.638396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:22.638396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:22.638396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:22.638396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:22.638396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:22.638396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.638396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:22.638396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:22.638396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.639135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:22.639135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:22.639135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.639135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:22.639135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:22.639135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.639135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:22.639135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:22.639135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:44:22.639135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:22.639135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:22.639135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.639135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:22.639135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:22.639135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:22.639135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.639135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:22.639135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:22.639135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:22.639651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:22.639651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:22.639651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:22.639651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:22.639651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:22.639651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:22.639651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:22.639651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:22.639651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:22.639651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:22.639651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.639651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:22.640149 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:22.640290 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:22.640389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:22.640506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:22.640799 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:22.640965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.641329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:22.641329 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:22.641329 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:22.641329 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:22.641329 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:22.641329 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:22.641329 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.641329 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:22.641856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.641856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:22.641856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:22.641856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:22.641856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.641856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:22.641856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:22.641856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:22.641856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.641856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:22.641856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:22.641856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:22.641856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:22.641856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:22.641856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:22.642326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:22.642326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:22.642326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:22.642326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:22.642326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:22.642326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:22.642326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:22.642326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.642326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:22.642565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.642565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:22.642565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:22.642565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:22.642565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.642565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:22.642565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:22.642565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:22.642565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:22.642565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.642565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:22.642863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:22.642863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:22.642863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.642863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:22.642863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:22.642863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:22.642863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:22.642863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:22.642863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:22.642863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:22.642863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:22.642863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:22.642863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:22.642863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:22.643191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:22.643191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:22.643191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.643191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:22.643370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.643370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:22.643370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:22.643370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:22.643370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:22.643370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:22.643370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.643370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:22.643370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:22.643370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.643370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:22.643370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:22.643370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.643370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:22.643370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:22.643370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.643370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:22.643796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:22.643796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:22.643796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:22.643796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.643796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:22.643796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:22.643796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:22.643796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.643796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:22.643796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:22.643796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:22.643796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:22.643796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:22.643796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:22.643796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:22.643796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:22.643796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:22.643796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:22.643796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:22.644102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:22.644102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:22.644102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.644102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:22.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:22.651142 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:22.651440 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:22.651646 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:22.651956 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:22.652111 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:22.652240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.652719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:22.652719 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:22.652719 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:22.652719 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:22.652719 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:22.652719 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:22.652719 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.652719 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:22.653587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.653587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:22.653587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:22.653587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:22.653587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.653587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:22.653587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:22.653587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.653587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:22.653587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:22.653587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:22.653587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.653928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:22.653928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:22.653928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:22.653928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:22.653928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:22.653928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:22.653928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:22.653928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:22.653928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:22.653928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:22.653928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:22.653928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:22.653928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:22.654240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.654240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:22.654604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.654604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:22.654604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:22.654604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:22.654604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:22.654604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:22.654604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.654604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:22.654604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:22.654604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.654604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:22.654604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:22.654604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.654604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:22.654604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:22.654604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:22.655040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:22.655040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:22.655040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.655040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:22.655040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:22.655040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:22.655040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:22.655040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:22.655040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:22.655040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:22.655040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:22.655040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:22.655040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:22.655040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:22.655040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:22.655040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:22.655040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:22.655040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:22.655040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:22.655640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:22.655640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:22.655640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:23.831703 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:23.832209 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:23.832209 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:23.832905 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:23.864528 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:23.864767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:23.865341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:23.865341 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:23.865341 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:23.865341 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:23.865341 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:23.865341 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:23.865341 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:23.865341 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:23.866075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:23.866075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:23.866075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:23.866075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:23.866075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:23.866075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:23.866075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:23.866075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:23.866075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:23.866075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:23.866075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:23.866075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:23.866588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:23.866588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:23.866588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:23.866588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:23.866588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:23.866588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:23.866588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:23.866588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:23.866588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:23.866588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:23.866588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:23.866588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:23.866588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:23.867019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:23.867019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:23.867019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:23.867019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:23.867019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:23.867019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:23.867019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:23.867019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:23.867019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:23.867019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:23.867019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:23.867019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:23.867019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:23.867019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:23.867019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:23.867656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:23.867656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:23.867656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:23.867656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:23.867656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:23.867656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:23.867656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:23.867656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:23.867656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:23.867656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:23.867656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:23.867656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:23.867656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:23.867656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:23.867656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:23.867656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:23.867656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:23.867656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:23.867656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:23.868349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:23.868349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:23.868349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:23.868349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:23.868349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:23.868349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:27.632279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:27.632279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:27.632279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:27.632279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:27.632279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:27.633432 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:27.633432 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:27.633432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.633432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:27.633432 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:27.633432 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:27.633432 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:27.633432 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:27.633432 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:27.633432 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.633432 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:27.633432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.633852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:27.633852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:27.633852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:27.633852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.633852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:27.633852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:27.633852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.633852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:27.633852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:27.633852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:27.633852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.633852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:27.633852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:27.633852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:27.634236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:27.634236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:27.634236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:27.634236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:27.634236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:27.634236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:27.634236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:27.634236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:27.634236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:27.634236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:27.634236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.634236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:27.634611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.634611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:27.634611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:27.634611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:27.634611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:27.634611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:27.634611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.634611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:27.634611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:27.634611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.634611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:27.634611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:27.634611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.634611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:27.634611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:27.634611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:27.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:27.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:27.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:27.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:27.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:27.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:27.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:27.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:27.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:27.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:27.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:27.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:27.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:27.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:27.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:27.635043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:27.635551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:27.635551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.635551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:27.635551 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:27.635551 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:27.635697 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:27.635697 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:44:27.636675 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:27.640682 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:27.640972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.641678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:27.641678 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:27.641678 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:27.641678 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:27.641678 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:27.641678 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:27.641678 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.641678 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:27.642392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.642392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:27.642392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:27.642392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:27.642392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.642392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:27.642392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:27.642392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:27.642392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.642392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:27.642392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:27.642392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:27.642392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:27.642392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:27.642392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:27.642825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:27.642825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:27.642825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:27.642825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:27.642825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:27.642825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:27.642825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:27.642825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.642825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:27.643079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.643079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:27.643079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:27.643079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:27.643079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.643079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:27.643079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:27.643079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:44:27.643079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:27.643079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:27.643079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.643079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:27.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:27.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:27.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:27.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:27.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:27.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:27.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:27.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:27.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:27.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:27.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:27.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:27.643420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:27.643757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:27.643757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:27.643757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.643757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:27.643757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.643757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:27.643757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:27.643757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:27.643757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:27.643757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:27.643757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.643757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:27.643757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:27.643757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.644095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:27.644095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:27.644095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.644095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:27.644095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:27.644095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.644095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:27.644095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:27.644095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:44:27.644095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:27.644095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:27.644095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.644095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:27.644095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:27.644095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:27.644095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.644095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:27.644095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:27.644095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:27.644593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:27.644593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:27.644593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:27.644593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:27.644593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:27.644593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:27.644593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:27.644593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:27.644593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:27.644593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:27.644593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.644593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:27.644593 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:27.644593 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:27.644939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:27.645802 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:27.645802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.646263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:27.646263 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:27.646263 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:27.646263 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:27.646263 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:27.646263 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:27.646263 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.646263 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:27.646977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.646977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:27.646977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:27.646977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:27.646977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.646977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:27.646977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:27.646977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:27.646977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.646977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:27.646977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:27.646977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:27.646977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:27.646977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:27.646977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:27.647362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:27.647362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:27.647362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:27.647362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:27.647362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:27.647362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:27.647362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:27.647362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.647362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:27.647732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.647732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:27.647732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:27.647732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:27.647732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.647732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:27.647732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:27.647732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:27.647732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:27.647732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.647732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:27.648023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:27.648023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:27.648023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.648023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:27.648023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:27.648023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:27.648023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:27.648023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:27.648023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:27.648023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:27.648023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:27.648023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:27.648023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:27.648023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:27.648477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:27.648477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:27.648477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.648477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:27.648477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.648477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:27.648477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:27.648477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:27.648477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:27.648477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:27.648477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.648477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:27.648477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:27.648477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:27.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:27.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:27.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:27.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:27.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:27.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:27.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:27.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:27.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:27.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:27.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:27.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:27.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:27.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:27.649344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:27.649344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:27.649344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:27.649344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:27.649344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:27.649344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:27.649344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:27.649344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:27.649344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:27.649344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.649344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:27.653107 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:27.653482 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:27.653823 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:27.654049 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:27.654438 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:27.655378 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:27.655378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.655578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:27.655578 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:27.655578 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:27.655578 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:27.655578 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:27.655578 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:27.655578 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.655578 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:27.656279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.656279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:27.656279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:27.656279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:27.656279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.656279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:27.656279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:27.656279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.656279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:27.656279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:27.656279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:27.656279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.656605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:27.656605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:27.656605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:27.656605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:27.656605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:27.656605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:27.656605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:27.656605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:27.656605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:27.656605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:27.656605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:27.656605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:27.656605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:27.656908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.656908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:27.657038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.657038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:27.657038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:27.657038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:27.657038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:27.657038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:27.657038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.657038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:27.657038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:27.657038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.657038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:27.657038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:27.657038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.657038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:27.657038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:27.657038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:27.657445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:27.657445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:27.657445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.657445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:27.657445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:27.657445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:27.657445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:27.657445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:27.657445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:27.657445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:27.657445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:27.657445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:27.657445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:27.657445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:27.657445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:27.657445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:27.657445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:27.657445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:27.657445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:27.657991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:27.657991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:27.657991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:28.832199 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:28.832800 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:28.832845 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:28.833692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:28.864604 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:28.864604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:28.864893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:28.864893 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:28.864893 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:28.864893 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:28.864893 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:28.864893 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:28.864893 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:28.864893 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:28.866008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:28.866008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:28.866008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:28.866008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:28.866008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:28.866008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:28.866008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:28.866008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:28.866008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:28.866008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:28.866008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:28.866008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:28.866425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:28.866425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:28.866425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:28.866425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:28.866425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:28.866425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:28.866425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:28.866425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:28.866425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:28.866425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:28.866425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:28.866425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:28.866425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:28.866774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:28.866774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:28.867430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:28.867430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:28.867430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:28.867430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:28.867430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:28.867430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:28.867430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:28.867430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:28.867430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:28.867430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:28.867430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:28.867430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:28.867430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:28.867430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:28.867430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:28.867430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:28.868006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:28.868006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:28.868006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:28.868006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:28.868006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:28.868006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:28.868006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:28.868006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:28.868006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:28.868006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:28.868006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:28.868006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:28.868006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:28.868006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:28.868006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:28.868006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:28.868006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:28.868006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:28.868006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:28.868452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:28.868452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:28.868452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:32.629097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:32.629510 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:32.629510 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:32.631181 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:32.631181 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:32.631181 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:32.631181 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:32.631181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.631360 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:32.631360 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:32.631360 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:32.631360 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:32.631360 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:32.631360 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:32.631360 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.631360 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:32.632161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.632161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:32.632161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:32.632161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:32.632161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.632161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:32.632161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:32.632161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.632161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:32.632161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:32.632161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:32.632161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.632678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:32.632678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:32.632678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:32.632678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:32.632678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:32.632678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:32.632678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:32.632678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:32.632678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:32.632678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:32.632678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:32.632678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:32.632678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:32.633348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.633348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:32.633348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.633348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:32.633348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:32.633348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:32.633348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:32.633348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:32.633348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.633348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:32.633348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:32.633348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.633348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:32.633348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:32.633348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:32.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:32.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:32.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:32.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:32.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:32.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:32.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:32.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:32.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:32.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:32.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:32.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:32.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:32.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:32.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:32.634555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:32.634555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:32.634555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:32.634555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:32.634555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.634555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:32.635608 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:32.635802 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:32.635888 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:32.635993 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:44:32.636875 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:32.636875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.637013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:32.637013 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:32.637013 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:32.637013 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:32.637013 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:32.637013 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:32.637013 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.637013 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:32.637583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.637583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:32.637583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:32.637583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:32.637583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.637583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:32.637583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:32.637583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:32.637583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.637583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:32.637583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:32.637583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:32.637583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:32.637583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:32.637583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:32.637817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:32.637817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:32.637817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:32.637817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:32.637817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:32.637817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:32.637817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:32.637817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.637817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:32.638279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.638279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:32.638279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:32.638279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:32.638279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.638279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:32.638279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:32.638279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:44:32.638279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:32.638279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:32.638279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.638279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:32.638863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:32.638863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:32.638863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.638863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:32.638863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:32.638863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:32.638863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:32.638863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:32.638863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:32.638863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:32.638863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:32.638863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:32.638863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:32.638863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:32.639592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:32.639592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:32.639592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.639592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:32.639592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.639592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:32.639592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:32.639592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:32.639592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:32.639592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:32.639592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.639592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:32.639592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:32.639592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.639914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:32.639914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:32.639914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.639914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:32.639914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:32.639914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.639914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:32.639914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:32.639914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:44:32.639914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:32.639914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:32.639914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.639914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:32.639914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:32.639914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:32.639914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.639914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:32.639914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:32.639914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:32.640213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:32.640213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:32.640213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:32.640213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:32.640213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:32.640213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:32.640213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:32.640213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:32.640213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:32.640213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:32.640213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.640213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:32.644032 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:32.644245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:32.644362 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:32.644456 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:32.645451 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:32.645451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.645641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:32.645641 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:32.645641 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:32.645641 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:32.645641 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:32.645641 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:32.645641 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.645641 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:32.646493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.646493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:32.646493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:32.646493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:32.646493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.646493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:32.646493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:32.646493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:32.646493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.646493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:32.646493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:32.646493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:32.646493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:32.646493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:32.646493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:32.647155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:32.647155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:32.647155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:32.647155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:32.647155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:32.647155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:32.647155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:32.647155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.647155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:32.647155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.647155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:32.647155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:32.647155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:32.647155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.647455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:32.647455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:32.647455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:32.647455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:32.647455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.647455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:32.647455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:32.647455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:32.647455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.647455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:32.647455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:32.647455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:32.647455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:32.647695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:32.647695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:32.647695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:32.647695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:32.647695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:32.647695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:32.647695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:32.647695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:32.647695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:32.647695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.647695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:32.647924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.647924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:32.647924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:32.647924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:32.647924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:32.647924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:32.647924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.647924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:32.647924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:32.647924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.647924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:32.647924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:32.647924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.647924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:32.647924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:32.647924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.647924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:32.648268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:32.648268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:32.648268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:32.648268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.648268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:32.648268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:32.648268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:32.648268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.648268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:32.648268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:32.648268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:32.648268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:32.648268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:32.648268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:32.648268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:32.648268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:32.648268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:32.648268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:32.648268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:32.648717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:32.648717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:32.648717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.648717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:32.653435 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:32.653685 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:32.653820 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:32.654000 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:32.655608 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:32.655608 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:32.655608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.655727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:32.655727 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:32.655727 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:32.655727 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:32.655727 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:32.655727 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:32.655727 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.655727 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:32.656382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.656382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:32.656382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:32.656382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:32.656382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.656382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:32.656382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:32.656382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.656382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:32.656382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:32.656382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:32.656382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.656707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:32.656707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:32.656707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:32.656707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:32.656707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:32.656707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:32.656707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:32.656707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:32.656707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:32.656707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:32.656707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:32.656707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:32.656707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:32.657078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.657078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:32.657078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.657078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:32.657078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:32.657078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:32.657078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:32.657078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:32.657078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.657078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:32.657078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:32.657078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.657078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:32.657078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:32.657078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.657620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:32.657620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:32.657620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:32.657620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:32.657620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:32.657620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.657620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:32.657620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:32.657620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:32.657620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:32.657620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:32.657620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:32.657620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:32.657620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:32.657620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:32.657620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:32.657620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:32.657620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:32.657620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:32.658162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:32.658162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:32.658162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:32.658162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:32.658162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:32.658162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:33.832708 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:33.833335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:33.833335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:33.833833 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:33.864001 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:33.864383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:33.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:33.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:33.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:33.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:33.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:33.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:33.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:33.864940 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:33.865883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:33.865883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:33.865883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:33.865883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:33.865883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:33.865883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:33.865883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:33.865883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:33.865883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:33.865883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:33.865883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:33.865883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:33.866184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:33.866184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:33.866184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:33.866184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:33.866184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:33.866184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:33.866184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:33.866184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:33.866184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:33.866184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:33.866184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:33.866184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:33.866184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:33.866464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:33.866464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:33.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:33.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:33.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:33.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:33.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:33.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:33.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:33.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:33.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:33.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:33.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:33.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:33.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:33.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:33.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:33.866934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:33.867327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:33.867327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:33.867327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:33.867327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:33.867327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:33.867327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:33.867327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:33.867327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:33.867327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:33.867327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:33.867327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:33.867327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:33.867327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:33.867327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:33.867327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:33.867327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:33.867327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:33.867327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:33.867327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:33.867715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:33.867715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:33.867715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:37.631522 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:37.632097 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:37.632315 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:37.633165 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:37.633349 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:37.633467 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:37.633619 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:37.633806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.634460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:37.634460 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:37.634460 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:37.634460 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:37.634460 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:37.634460 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:37.634460 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.634460 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:37.635107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.635107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:37.635107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:37.635107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:37.635107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.635107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:37.635107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:37.635107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.635107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:37.635107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:37.635107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:37.635107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.635685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:37.635685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:37.635685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:37.635685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:37.635685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:37.635685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:37.635685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:37.635685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:37.635685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:37.635685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:37.635685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:37.635685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:37.635685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:37.636074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.636074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:37.636074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.636074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:37.636074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:37.636074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:37.636074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:37.636074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:37.636074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.636074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:37.636074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:37.636074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.636074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:37.636074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:37.636074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.636532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:37.636532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:37.636532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.636532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:37.636532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:37.636532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.636532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:37.636532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:37.636532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:37.636532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.636532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:37.636532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:37.636532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:37.636532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:37.636532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:37.636532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:37.636532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:37.636532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:37.636532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:37.637052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:37.637052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:37.637052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:37.637052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:37.637052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.637052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:37.637052 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:37.637052 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:37.637052 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:37.637052 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:44:37.637355 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:37.637355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.637355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:37.637355 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:37.637355 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:37.637355 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:37.637355 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:37.637355 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:37.637355 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.637355 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:37.637849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.637849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:37.637849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:37.637849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:37.637849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.637849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:37.637849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:37.637849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:37.637849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.637849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:37.637849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:37.637849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:37.637849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:37.637849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:37.637849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:37.638375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:37.638375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:37.638375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:37.638375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:37.638375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:37.638375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:37.638375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:37.638375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.638375 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:37.638693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.638693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:37.638693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:37.638693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:37.638693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.638693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:37.638693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:37.638693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:44:37.638693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:37.638693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:37.638693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.638693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:37.639012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:37.639012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:37.639012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.639012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:37.639012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:37.639012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:37.639012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:37.639012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:37.639012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:37.639012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:37.639012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:37.639012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:37.639012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:37.639012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:37.639450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:37.639450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:37.639450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.639450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:37.639450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.639450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:37.639450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:37.639450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:37.639450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:37.639450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:37.639450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.639450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:37.639450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:37.639450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.639885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:37.639885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:37.639885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.639885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:37.639885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:37.639885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.639885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:37.639885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:37.639885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:44:37.639885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:37.639885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:37.639885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.639885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:37.639885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:37.639885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:37.639885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.639885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:37.639885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:37.639885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:37.640466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:37.640466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:37.640466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:37.640466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:37.640466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:37.640466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:37.640466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:37.640466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:37.640466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:37.640466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:37.640466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.640466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:37.644427 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:37.644785 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:37.644949 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:37.645099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:37.646199 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:37.646469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.647117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:37.647117 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:37.647117 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:37.647117 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:37.647117 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:37.647117 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:37.647117 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.647117 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:37.648009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.648009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:37.648009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:37.648009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:37.648009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.648009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:37.648009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:37.648009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:37.648009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.648009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:37.648009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:37.648009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:37.648009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:37.648009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:37.648009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:37.648512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:37.648512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:37.648512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:37.648512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:37.648512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:37.648512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:37.648512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:37.648512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.648512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:37.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:37.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:37.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:37.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:37.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:37.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:37.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:37.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:37.649469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:37.649469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:37.649469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.649469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:37.649469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:37.649469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:37.649469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:37.649469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:37.649469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:37.649469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:37.649469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:37.649469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:37.649469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:37.649469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:37.649957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:37.649957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:37.649957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.649957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:37.650308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.650308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:37.650308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:37.650308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:37.650308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:37.650308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:37.650308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.650308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:37.650308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:37.650308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.650308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:37.650308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:37.650308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.650308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:37.650308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:37.650308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.650308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:37.651136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:37.651136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:37.651136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:37.651136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.651136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:37.651136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:37.651136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:37.651136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.651136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:37.651136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:37.651136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:37.651136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:37.651136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:37.651136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:37.651136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:37.651136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:37.651136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:37.651136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:37.651136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:37.651605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:37.651605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:37.651605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.651605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:37.653864 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:37.654492 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:37.654654 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:37.654886 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:37.655183 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:37.655435 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:37.655612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.656114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:37.656114 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:37.656114 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:37.656114 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:37.656114 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:37.656114 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:37.656114 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.656114 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:37.657066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.657066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:37.657066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:37.657066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:37.657066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.657066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:37.657066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:37.657066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.657066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:37.657066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:37.657066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:37.657066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.657485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:37.657485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:37.657485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:37.657485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:37.657485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:37.657485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:37.657485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:37.657485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:37.657485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:37.657485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:37.657485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:37.657485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:37.657485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:37.658087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.658087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:37.658087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.658087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:37.658087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:37.658087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:37.658087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:37.658087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:37.658087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.658087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:37.658087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:37.658087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.658087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:37.658087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:37.658087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.658568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:37.658568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:37.658568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:37.658568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:37.658568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:37.658568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.658568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:37.658568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:37.658568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:37.658568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:37.658568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:37.658568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:37.658568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:37.658568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:37.658568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:37.658568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:37.658568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:37.658568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:37.658568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:37.659201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:37.659201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:37.659201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:37.659201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:37.659201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:37.659201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:38.833144 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:38.833636 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:38.833782 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:38.834528 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:38.867410 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:38.867731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:38.868460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:38.868460 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:38.868460 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:38.868460 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:38.868460 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:38.868460 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:38.868460 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:38.868460 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:38.869310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:38.869310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:38.869310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:38.869310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:38.869310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:38.869310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:38.869310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:38.869310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:38.869310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:38.869310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:38.869310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:38.869310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:38.869641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:38.869641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:38.869641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:38.869641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:38.869641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:38.869641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:38.869641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:38.869641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:38.869641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:38.869641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:38.869641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:38.869641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:38.869641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:38.869856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:38.869856 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:38.870189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:38.870189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:38.870189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:38.870189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:38.870189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:38.870189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:38.870189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:38.870189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:38.870189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:38.870189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:38.870189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:38.870189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:38.870189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:38.870189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:38.870189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:38.870189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:38.870613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:38.870613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:38.870613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:38.870613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:38.870613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:38.870613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:38.870613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:38.870613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:38.870613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:38.870613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:38.870613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:38.870613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:38.870613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:38.870613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:38.870613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:38.870613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:38.870613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:38.870613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:38.870613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:38.870920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:38.870920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:38.870920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:42.632120 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:42.632724 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:42.632724 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:42.633362 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:42.633566 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:42.633783 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:42.633963 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:42.634117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.634573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:42.634573 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:42.634573 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:42.634573 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:42.634573 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:42.634573 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:42.634573 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.634573 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:42.635192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.635192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:42.635192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:42.635192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:42.635192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.635192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:42.635192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:42.635192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.635192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:42.635192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:42.635192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:42.635192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.635434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:42.635434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:42.635434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:42.635434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:42.635434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:42.635434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:42.635434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:42.635434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:42.635434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:42.635434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:42.635434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:42.635434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:42.635434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:42.635685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.635685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:42.636080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.636080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:42.636080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:42.636080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:42.636080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:42.636080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:42.636080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.636080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:42.636080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:42.636080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.636080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:42.636080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:42.636080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.636080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:42.636080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:42.636080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.637038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:42.637038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:42.637038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.637038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:42.637038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:42.637038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:42.637038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.637038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:42.637038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:42.637038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:42.637038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:42.637038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:42.637038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:42.637038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:42.637038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:42.637038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:42.637038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:42.637038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:42.637038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:42.637561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:42.637561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.637561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:42.637561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:42.637561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:42.637561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:42.637561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:44:42.637561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:42.637722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.637722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:42.637722 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:42.637722 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:42.637722 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:42.637722 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:42.637722 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:42.637722 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.637722 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:42.638156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.638156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:42.638156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:42.638156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:42.638156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.638156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:42.638156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:42.638156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:42.638156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.638156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:42.638156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:42.638156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:42.638156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:42.638156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:42.638156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:42.638986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:42.638986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:42.638986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:42.638986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:42.638986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:42.638986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:42.638986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:42.638986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.638986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:42.638986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.638986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:42.638986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:42.638986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:42.638986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.639242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:42.639242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:42.639242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:44:42.639242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:42.639242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:42.639242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.639242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:42.639242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:42.639242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:42.639242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.639242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:42.639242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:42.639242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:42.639486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:42.639486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:42.639486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:42.639486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:42.639486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:42.639486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:42.639486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:42.639486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:42.639486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:42.639486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:42.639486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.639486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:42.639708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.639708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:42.639708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:42.639708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:42.639708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:42.639708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:42.639708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.639708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:42.639708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:42.639708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.639708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:42.639708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:42.639708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.639708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:42.639708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:42.639708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.639942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:42.639942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:42.639942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:44:42.639942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:42.639942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:42.639942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.639942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:42.639942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:42.639942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:42.639942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.639942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:42.639942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:42.639942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:42.639942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:42.639942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:42.639942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:42.639942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:42.639942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:42.640243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:42.640243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:42.640243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:42.640243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:42.640243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:42.640243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.640243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:42.644843 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:42.645069 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:42.645176 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:42.645305 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:42.645651 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:42.645889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.646454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:42.646454 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:42.646454 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:42.646454 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:42.646454 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:42.646454 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:42.646454 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.646454 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:42.647030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.647030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:42.647030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:42.647030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:42.647030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.647030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:42.647030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:42.647030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:42.647030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.647030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:42.647030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:42.647030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:42.647030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:42.647030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:42.647030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:42.647916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:42.647916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:42.647916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:42.647916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:42.647916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:42.647916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:42.647916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:42.647916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.647916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:42.647916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.647916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:42.647916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:42.647916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:42.647916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.648368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:42.648368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:42.648368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:42.648368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:42.648368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.648368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:42.648368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:42.648368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:42.648368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.648368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:42.648368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:42.648368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:42.648368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:42.648586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:42.648586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:42.648586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:42.648586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:42.648586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:42.648586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:42.648586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:42.648586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:42.648586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:42.648586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.648586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:42.648586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.648775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:42.648775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:42.648775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:42.648775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:42.648775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:42.648775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.648775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:42.648775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:42.648775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.648775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:42.648775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:42.648775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.648775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:42.648775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:42.648775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.648775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:42.648775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:42.648775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:42.648775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.649049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:42.654505 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:42.654900 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:42.655030 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:42.655157 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:42.655515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:42.655780 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:42.655962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.656608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:42.656608 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:42.656608 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:42.656608 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:42.656608 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:42.656608 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:42.656608 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.656608 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:42.657287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.657287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:42.657287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:42.657287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:42.657287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.657287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:42.657287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:42.657287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.657287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:42.657287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:42.657287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:42.657287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.658188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:42.658188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:42.658188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:42.658188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:42.658188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:42.658188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:42.658188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:42.658188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:42.658188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:42.658188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:42.658188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:42.658188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:42.658188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:42.659114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.659114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:42.659114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.659114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:42.659114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:42.659114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:42.659114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:42.659114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:42.659114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.659114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:42.659114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:42.659114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.659114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:42.659114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:42.659114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.660364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:42.660364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:42.660364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:42.660364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:42.660364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:42.660364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.660364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:42.660364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:42.660364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:42.660364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:42.660364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:42.660364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:42.660364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:42.660364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:42.660364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:42.660364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:42.660364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:42.660364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:42.660364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:42.661058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:42.661058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:42.661058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:42.661058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:42.661058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:42.661058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:43.833775 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:43.834528 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:43.834528 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:43.835386 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:43.872830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:43.873127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:43.873681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:43.873681 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:43.873681 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:43.873681 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:43.873681 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:43.873681 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:43.873681 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:43.873681 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:43.874685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:43.874685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:43.874685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:43.874685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:43.874685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:43.874685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:43.874685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:43.874685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:43.874685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:43.874685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:43.874685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:43.874685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:43.875531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:43.875531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:43.875531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:43.875531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:43.875531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:43.875531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:43.875531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:43.875531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:43.875531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:43.875531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:43.875531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:43.875531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:43.875531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:43.875904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:43.875904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:43.875904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:43.875904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:43.875904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:43.875904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:43.875904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:43.875904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:43.875904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:43.875904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:43.875904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:43.875904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:43.875904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:43.875904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:43.875904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:43.876364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:43.876364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:43.876364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:43.876364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:43.876364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:43.876364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:43.876364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:43.876364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:43.876364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:43.876364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:43.876364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:43.876364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:43.876364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:43.876364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:43.876364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:43.876364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:43.876364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:43.876364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:43.876364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:43.878004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:43.878004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:43.878004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:43.878004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:43.878004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:43.878004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:47.632591 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:47.633058 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:47.633286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:47.634199 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:47.634469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:47.634687 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:47.634848 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:47.634994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.635684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:47.635684 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:47.635684 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:47.635684 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:47.635684 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:47.635684 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:47.635684 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.635684 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:47.636791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.636791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:47.636791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:47.636791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:47.636791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.636791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:47.636791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:47.636791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.636791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:47.636791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:47.636791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:47.636791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.637154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:47.637154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:47.637154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:47.637154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:47.637154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:47.637154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:47.637154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:47.637154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:47.637154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:47.637154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:47.637154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:47.637154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:47.637154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:47.637419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.637419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:47.637419 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:47.638446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.638446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:47.638446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:47.638446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:47.638446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:47.638446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:47.638446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.638446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:47.638446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:47.638446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.638446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:47.638446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:47.638446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.638446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:47.638446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:47.638446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.639114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:47.639114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:47.639114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.639114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:47.639114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:47.639114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:47.639114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.639114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:47.639114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:47.639114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:47.639114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:47.639114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:47.639114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:47.639114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:47.639114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:47.639114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:47.639114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:47.639114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:47.639114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:47.639749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:47.639749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.639749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:47.639749 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:47.639749 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:47.639749 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:44:47.639749 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:47.639749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.640237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:47.640237 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:47.640237 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:47.640237 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:47.640237 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:47.640237 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:47.640237 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.640237 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:47.641155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.641155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:47.641155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:47.641155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:47.641155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.641155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:47.641155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:47.641155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:47.641155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.641155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:47.641155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:47.641155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:47.641155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:47.641155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:47.641155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:47.641455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:47.641455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:47.641455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:47.641455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:47.641455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:47.641455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:47.641455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:47.641455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.641455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:47.642094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.642094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:47.642094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:47.642094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:47.642094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.642094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:47.642094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:47.642094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:44:47.642094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:47.642094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:47.642094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.642094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:47.642351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:47.642351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:47.642351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.642351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:47.642351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:47.642351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:47.642351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:47.642351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:47.642351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:47.642351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:47.642351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:47.642351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:47.642351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:47.642351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:47.642626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:47.642626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:47.642626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.642626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:47.643393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.643393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:47.643393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:47.643393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:47.643393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:47.643393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:47.643393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.643393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:47.643393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:47.643393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.643393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:47.643393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:47.643393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.643393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:47.643393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:47.643393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.643729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:47.643729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:47.643729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:44:47.643729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:47.643729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:47.643729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.643729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:47.643729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:47.643729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:47.643729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.643729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:47.643729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:47.643729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:47.643729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:47.643729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:47.643729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:47.643729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:47.643729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:47.644009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:47.644009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:47.644009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:47.644009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:47.644009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:47.644009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.644009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:47.645233 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:47.645628 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:47.645795 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:47.645951 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:47.646411 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:47.646666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.647494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:47.647494 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:47.647494 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:47.647494 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:47.647494 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:47.647494 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:47.647494 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.647494 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:47.648106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.648106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:47.648106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:47.648106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:47.648106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.648106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:47.648106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:47.648106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:47.648106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.648106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:47.648106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:47.648106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:47.648106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:47.648106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:47.648106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:47.648581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:47.648581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:47.648581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:47.648581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:47.648581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:47.648581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:47.648581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:47.648581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.648581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:47.649098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.649098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:47.649098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:47.649098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:47.649098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.649098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:47.649098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:47.649098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:47.649098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:47.649098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.649098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:47.649350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:47.649350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:47.649350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.649350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:47.649350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:47.649350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:47.649350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:47.649350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:47.649350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:47.649350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:47.649350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:47.649350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:47.649350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:47.649350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:47.649598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:47.649598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:47.649598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.649598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:47.650222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.650222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:47.650222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:47.650222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:47.650222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:47.650222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:47.650222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.650222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:47.650222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:47.650222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.650222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:47.650222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:47.650222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.650222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:47.650222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:47.650222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.650222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:47.650623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:47.650623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:47.650623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:47.650623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.650623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:47.650623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:47.650623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:47.650623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.650623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:47.650623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:47.650623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:47.650623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:47.650623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:47.650623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:47.650623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:47.650623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:47.650623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:47.650623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:47.650623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:47.650902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:47.650902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:47.650902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.650902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:47.655003 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:47.655483 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:47.655799 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:47.655799 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:47.656321 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:47.656596 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:47.656781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.657235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:47.657235 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:47.657235 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:47.657235 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:47.657235 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:47.657235 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:47.657235 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.657235 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:47.657978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.657978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:47.657978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:47.657978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:47.657978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.657978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:47.657978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:47.657978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.657978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:47.657978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:47.657978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:47.657978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.658241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:47.658241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:47.658241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:47.658241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:47.658241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:47.658241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:47.658241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:47.658241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:47.658241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:47.658241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:47.658241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:47.658241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:47.658241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:47.658496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.658496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:47.658803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.658803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:47.658803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:47.658803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:47.658803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:47.658803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:47.658803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.658803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:47.658803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:47.658803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.658803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:47.658803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:47.658803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.658803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:47.658803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:47.658803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:47.659408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:47.659408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:47.659408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.659408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:47.659408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:47.659408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:47.659408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:47.659408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:47.659408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:47.659408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:47.659408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:47.659408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:47.659408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:47.659408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:47.659408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:47.659408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:47.659408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:47.659408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:47.659408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:47.660007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:47.660007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:47.660007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:48.834357 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:48.834844 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:48.834880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:48.835658 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:48.876819 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:48.876985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:48.877554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:48.877554 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:48.877554 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:48.877554 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:48.877554 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:48.877554 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:48.877554 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:48.877554 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:48.878723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:48.878723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:48.878723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:48.878723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:48.878723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:48.878723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:48.878723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:48.878723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:48.878723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:48.878723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:48.878723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:48.878723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:48.879042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:48.879042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:48.879042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:48.879042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:48.879042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:48.879042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:48.879042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:48.879042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:48.879042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:48.879042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:48.879042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:48.879042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:48.879042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:48.879300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:48.879300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:48.880448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:48.880448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:48.880448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:48.880448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:48.880448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:48.880448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:48.880448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:48.880448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:48.880448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:48.880448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:48.880448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:48.880448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:48.880448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:48.880448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:48.880448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:48.880448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:48.880794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:48.880794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:48.880794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:48.880794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:48.880794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:48.880794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:48.880794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:48.880794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:48.880794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:48.880794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:48.880794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:48.880794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:48.880794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:48.880794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:48.880794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:48.880794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:48.880794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:48.880794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:48.880794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:48.881114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:48.881114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:48.881114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:52.633061 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:52.633609 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:52.633662 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:52.634445 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:52.634725 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:52.634906 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:52.635063 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:52.635206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.635865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:52.635865 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:52.635865 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:52.635865 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:52.635865 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:52.635865 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:52.635865 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.635865 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:52.636888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.636888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:52.636888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:52.636888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:52.636888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.636888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:52.636888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:52.636888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.636888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:52.636888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:52.636888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:52.636888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.637289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:52.637289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:52.637289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:52.637289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:52.637289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:52.637289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:52.637289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:52.637289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:52.637289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:52.637289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:52.637289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:52.637289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:52.637289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:52.637703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.637703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:52.637936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.637936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:52.637936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:52.637936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:52.637936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:52.637936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:52.637936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.637936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:52.637936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:52.637936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.637936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:52.637936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:52.637936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.637936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:52.637936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:52.637936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.638393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:52.638393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:52.638393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.638393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:52.638393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:52.638393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:52.638393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.638393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:52.638393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:52.638393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:52.638393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:52.638393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:52.638393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:52.638393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:52.638393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:52.638393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:52.638393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:52.638393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:52.638393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:52.638962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:52.638962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.638962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:52.638962 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:52.639081 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:52.639120 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:52.639231 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:44:52.639646 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:52.639806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.640265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:52.640265 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:52.640265 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:52.640265 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:52.640265 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:52.640265 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:52.640265 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.640265 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:52.641018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.641018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:52.641018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:52.641018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:52.641018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.641018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:52.641018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:52.641018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:52.641018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.641018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:52.641018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:52.641018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:52.641018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:52.641018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:52.641018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:52.641494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:52.641494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:52.641494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:52.641494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:52.641494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:52.641494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:52.641494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:52.641494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.641494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:52.641861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.641861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:52.641861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:52.641861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:52.641861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.641861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:52.641861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:52.641861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:44:52.641861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:52.641861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:52.641861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.641861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:52.642263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:52.642263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:52.642263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.642263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:52.642263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:52.642263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:52.642263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:52.642263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:52.642263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:52.642263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:52.642263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:52.642263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:52.642263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:52.642263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:52.642573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:52.642573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:52.642573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.642573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:52.642934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.642934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:52.642934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:52.642934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:52.642934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:52.642934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:52.642934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.642934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:52.642934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:52.642934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.642934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:52.642934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:52.642934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.642934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:52.642934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:52.642934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.643432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:52.643432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:52.643432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:44:52.643432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:52.643432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:52.643432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.643432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:52.643432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:52.643432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:52.643432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.643432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:52.643432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:52.643432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:52.643432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:52.643432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:52.643432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:52.643432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:52.643432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:52.643934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:52.643934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:52.643934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:52.643934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:52.643934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:52.643934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.643934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:52.645695 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:52.645959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:52.646070 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:52.646194 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:52.646566 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:52.646770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.647264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:52.647264 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:52.647264 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:52.647264 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:52.647264 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:52.647264 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:52.647264 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.647264 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:52.648020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.648020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:52.648020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:52.648020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:52.648020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.648020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:52.648020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:52.648020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:52.648020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.648020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:52.648020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:52.648020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:52.648020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:52.648020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:52.648020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:52.648494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:52.648494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:52.648494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:52.648494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:52.648494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:52.648494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:52.648494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:52.648494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.648494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:52.648906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.648906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:52.648906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:52.648906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:52.648906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.648906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:52.648906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:52.648906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:52.648906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:52.648906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.648906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:52.649153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:52.649153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:52.649153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.649153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:52.649153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:52.649153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:52.649153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:52.649153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:52.649153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:52.649153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:52.649153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:52.649153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:52.649153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:52.649153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:52.649567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:52.649567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:52.649567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.649567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:52.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:52.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:52.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:52.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:52.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:52.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:52.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:52.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:52.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:52.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:52.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:52.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:52.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:52.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:52.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:52.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:52.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:52.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:52.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:52.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:52.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:52.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:52.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:52.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:52.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:52.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:52.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:52.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:52.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:52.651028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:52.651028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:52.651028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.651028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:52.655456 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:52.655720 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:52.655828 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:52.655980 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:52.656356 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:52.656554 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:52.656733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.657133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:52.657133 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:52.657133 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:52.657133 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:52.657133 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:52.657133 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:52.657133 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.657133 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:52.657925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.657925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:52.657925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:52.657925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:52.657925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.657925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:52.657925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:52.657925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.657925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:52.657925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:52.657925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:52.657925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.658182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:52.658182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:52.658182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:52.658182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:52.658182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:52.658182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:52.658182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:52.658182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:52.658182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:52.658182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:52.658182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:52.658182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:52.658182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:52.658457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.658457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:52.658912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.658912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:52.658912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:52.658912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:52.658912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:52.658912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:52.658912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.658912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:52.658912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:52.658912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.658912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:52.658912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:52.658912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.658912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:52.658912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:52.658912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:52.659234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:52.659234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:52.659234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.659234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:52.659234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:52.659234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:52.659234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:52.659234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:52.659234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:52.659234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:52.659234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:52.659234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:52.659234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:52.659234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:52.659234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:52.659234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:52.659234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:52.659234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:52.659234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:52.659597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:52.659597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:52.659597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:53.834750 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:53.835421 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:53.835421 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:53.835903 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:53.844600 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=242 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:44:53.871773 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:53.872008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:53.872608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:53.872608 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:53.872608 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:53.872608 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:53.872608 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:53.872608 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:53.872608 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:53.872608 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:53.873352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:53.873352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:53.873352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:53.873352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:53.873352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:53.873352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:53.873352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:53.873352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:53.873352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:53.873352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:53.873352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:53.873352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:53.873666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:53.873666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:53.873666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:53.873666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:53.873666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:53.873666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:53.873666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:53.873666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:53.873666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:53.873666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:53.873666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:53.873666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:53.873666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:53.873946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:53.873946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:53.874056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:53.874056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:53.874056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:53.874056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:53.874056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:53.874056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:53.874056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:53.874056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:53.874056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:53.874056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:53.874056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:53.874056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:53.874056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:53.874056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:53.874056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:53.874056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:53.874701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:53.874701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:53.874701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:53.874701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:53.874701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:53.874701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:53.874701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:53.874701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:53.874701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:53.874701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:53.874701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:53.874701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:53.874701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:53.874701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:53.874701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:53.874701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:53.874701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:53.874701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:53.874701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:53.875113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:53.875113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:53.875113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:57.633632 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:57.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:57.634123 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:57.635058 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:57.635058 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:57.635160 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:57.635282 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:57.635438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.635951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:57.635951 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:57.635951 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:57.635951 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:57.635951 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:57.635951 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:57.635951 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.635951 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:57.636769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.636769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:57.636769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:57.636769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:57.636769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.636769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:57.636769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:57.636769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.636769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:57.636769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:57.636769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:57.636769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:57.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:57.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:57.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:57.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:57.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:57.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:57.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:57.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:57.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:57.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:57.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:57.637102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:57.637593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.637593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:57.637717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.637717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:57.637717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:57.637717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:57.637717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:57.637717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:57.637717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.637717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:57.637717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:57.637717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.637717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:57.637717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:57.637717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.637717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:57.637717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:57.637717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.638127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:44:57.638127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:57.638127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.638127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:57.638127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:57.638127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:57.638127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.638127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:57.638127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:57.638127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:57.638127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:57.638127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:57.638127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:57.638127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:57.638127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:57.638127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:57.638127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:57.638127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:57.638127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:57.638574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:57.638574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.638574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:57.639020 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:57.641421 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:57.641621 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:57.641772 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:44:57.642664 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:57.642664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.643112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:57.643112 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:57.643112 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:57.643112 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:57.643112 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:57.643112 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:57.643112 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.643112 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:57.643944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.643944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:57.643944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:57.643944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:57.643944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.643944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:57.643944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:57.643944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:57.643944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.643944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:57.643944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:57.643944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:57.643944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:57.643944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:57.643944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:57.644244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:57.644244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:57.644244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:57.644244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:57.644244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:57.644244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:57.644244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:57.644244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.644244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:57.644595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.644595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:57.644595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:57.644595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:57.644595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.644595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:57.644595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:57.644595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:44:57.644595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:57.644595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:57.644595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.644595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:57.644862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:57.644862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:57.644862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.644862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:57.644862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:57.644862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:57.644862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:57.644862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:57.644862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:57.644862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:57.644862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:57.644862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:57.644862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:57.644862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:57.645146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:57.645146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:57.645146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.645146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:57.645643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.645643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:57.645643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:57.645643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:57.645643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:57.645643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:57.645643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.645643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:57.645643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:57.645643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.645643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:57.645643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:57.645643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.645643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:57.645643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:57.645643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.646108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:44:57.646108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:44:57.646108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:44:57.646108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:57.646108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:57.646108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.646108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:57.646108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:57.646108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:57.646108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.646108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:57.646108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:57.646108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:57.646108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:57.646108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:57.646108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:57.646108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:57.646108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:57.646513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:57.646513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:57.646513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:57.646513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:57.646513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:57.646513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.646513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:57.646513 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:57.646513 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:57.646513 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:57.646835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:57.647138 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:57.647522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.648243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:57.648243 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:57.648243 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:57.648243 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:57.648243 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:57.648243 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:57.648243 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.648243 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:57.648988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.648988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:44:57.648988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:57.648988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:57.648988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.648988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:57.648988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:57.648988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:44:57.648988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.648988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:44:57.648988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:57.648988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:44:57.648988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:57.648988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:44:57.648988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:44:57.649309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:57.649309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:44:57.649309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:57.649309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:44:57.649309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:44:57.649309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:57.649309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:44:57.649309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.649309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:44:57.649840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.649840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:57.649840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:57.649840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:57.649840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.649840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:57.649840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:57.649840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:57.649840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:57.649840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.649840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:57.650203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:57.650203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:57.650203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.650203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:57.650203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:57.650203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:57.650203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:57.650203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:57.650203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:57.650203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:57.650203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:57.650203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:57.650203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:57.650203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:57.650715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:57.650715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:57.650715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.650715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:57.650846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.650846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:57.650846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:57.650846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:57.650846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:57.650846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:57.650846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.650846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:57.650846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:57.650846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.650846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:57.650846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:57.650846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.650846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:57.650846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:57.650846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.650846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:44:57.651331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:44:57.651331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:44:57.651331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:44:57.651331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.651331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:57.651331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:57.651331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:57.651331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.651331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:57.651331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:57.651331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:57.651331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:57.651331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:57.651331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:57.651331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:57.651331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:57.651331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:57.651331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:57.651331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:57.651936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:57.651936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:57.651936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.651936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:57.655861 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:57.656111 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:57.656222 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:57.656434 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:57.657024 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:57.657368 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:57.657627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.658271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:57.658271 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:57.658271 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:57.658271 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:57.658271 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:57.658271 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:57.658271 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.658271 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:57.659277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.659277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:57.659277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:57.659277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:57.659277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.659277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:57.659277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:57.659277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.659277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:57.659277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:57.659277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:57.659277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.659643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:57.659643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:57.659643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:57.659643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:57.659643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:57.659643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:57.659643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:57.659643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:57.659643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:57.659643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:57.659643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:57.659643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:57.659643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:57.660202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.660202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:57.660424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.660424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:57.660424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:57.660424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:57.660424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:57.660424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:57.660424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.660424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:57.660424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:57.660424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.660424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:57.660424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:57.660424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.660424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:57.660424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:57.660424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:57.660921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:44:57.660921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:57.660921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.660921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:57.660921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:57.660921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:57.660921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:57.660921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:57.660921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:57.660921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:57.660921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:57.660921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:57.660921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:57.660921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:57.660921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:57.660921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:57.660921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:57.660921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:57.660921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:57.661596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:57.661596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:57.661596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:44:58.835183 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:44:58.835676 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:44:58.835676 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:44:58.836191 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:44:58.869274 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:44:58.869635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:58.870303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:44:58.870303 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:58.870303 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:44:58.870303 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:44:58.870303 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:58.870303 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:44:58.870303 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:58.870303 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:44:58.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:58.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:44:58.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:58.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:44:58.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:58.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:58.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:58.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:58.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:58.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:58.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:44:58.871208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:58.871488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:44:58.871488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:58.871488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:44:58.871488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:58.871488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:44:58.871488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:44:58.871488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:58.871488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:44:58.871488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:58.871488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:44:58.871488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:44:58.871488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:58.871488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:44:58.871857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:58.871857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:44:58.872176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:58.872176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:44:58.872176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:44:58.872176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:44:58.872176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:44:58.872176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:44:58.872176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:58.872176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:44:58.872176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:44:58.872176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:58.872176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:44:58.872176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:44:58.872176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:44:58.872176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:44:58.872176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:44:58.872176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:44:58.872649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:44:58.872649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:44:58.872649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:58.872649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:58.872649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:44:58.872649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:44:58.872649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:44:58.872649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:44:58.872649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:44:58.872649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:44:58.872649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:44:58.872649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:44:58.872649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:44:58.872649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:44:58.872649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:44:58.872649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:44:58.872649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:44:58.872649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:44:58.872649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:44:58.873198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:44:58.873198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:44:58.873198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:02.634280 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:02.634935 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:02.634935 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:02.635704 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:02.635907 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:02.636129 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:02.636231 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:02.636408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.637080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:02.637080 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:02.637080 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:02.637080 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:02.637080 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:02.637080 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:02.637080 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.637080 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:02.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:02.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:02.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:02.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:02.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:02.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:02.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:02.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:02.637965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.638205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:02.638205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:02.638205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:02.638205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:02.638205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:02.638205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:02.638205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:02.638205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:02.638205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:02.638205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:02.638205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:02.638205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:02.638205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:02.638441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.638441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:02.638906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.638906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:02.638906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:02.638906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:02.638906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:02.638906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:02.638906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.638906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:02.638906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:02.638906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.638906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:02.638906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:02.638906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.638906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:02.638906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:02.638906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.639188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:02.639188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:02.639188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.639188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:02.639188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:02.639188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:02.639188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.639188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:02.639188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:02.639188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:02.639188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:02.639188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:02.639188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:02.639188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:02.639188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:02.639188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:02.639188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:02.639188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:02.639188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:02.639507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:02.639507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.639507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:02.639507 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:02.639833 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:02.639987 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:02.640121 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:45:02.640556 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:02.640773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.641318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:02.641318 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:02.641318 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:02.641318 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:02.641318 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:02.641318 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:02.641318 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.641318 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:02.641946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.641946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:02.641946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:02.641946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:02.641946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.641946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:02.641946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:02.641946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:02.641946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.641946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:02.641946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:02.641946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:02.641946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:02.641946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:02.641946 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:02.642228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:02.642228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:02.642228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:02.642228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:02.642228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:02.642228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:02.642228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:02.642228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.642228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:02.642676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.642676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:02.642676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:02.642676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:02.642676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.642676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:02.642676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:02.642676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:45:02.642676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:02.642676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:02.642676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.642676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:02.642907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:02.642907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:02.642907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.642907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:02.642907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:02.642907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:02.642907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:02.642907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:02.642907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:02.642907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:02.642907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:02.642907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:02.642907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:02.642907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:02.643144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:02.643144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:02.643144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.643144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:02.643534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.643534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:02.643534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:02.643534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:02.643534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:02.643534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:02.643534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.643534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:02.643534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:02.643534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.643534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:02.643534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:02.643534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.643534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:02.643534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:02.643534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.643818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:02.643818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:02.643818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:45:02.643818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:02.643818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:02.643818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.643818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:02.643818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:02.643818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:02.643818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.643818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:02.643818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:02.643818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:02.643818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:02.643818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:02.643818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:02.643818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:02.643818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:02.644078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:02.644078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:02.644078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:02.644078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:02.644078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:02.644078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.644078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:02.646347 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:02.646680 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:02.646830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:02.646996 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:02.647415 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:02.647788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.648306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:02.648306 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:02.648306 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:02.648306 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:02.648306 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:02.648306 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:02.648306 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.648306 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:02.648991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.648991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:02.648991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:02.648991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:02.648991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.648991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:02.648991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:02.648991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:02.648991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.648991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:02.648991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:02.648991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:02.648991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:02.648991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:02.648991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:02.649329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:02.649329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:02.649329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:02.649329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:02.649329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:02.649329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:02.649329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:02.649329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.649329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:02.649695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.649695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:02.649695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:02.649695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:02.649695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.649695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:02.649695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:02.649695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:02.649695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:02.649695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.649695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:02.649918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:02.649918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:02.649918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.649918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:02.649918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:02.649918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:02.649918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:02.649918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:02.649918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:02.649918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:02.649918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:02.649918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:02.649918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:02.649918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:02.650143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:02.650143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:02.650143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.650143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:02.650469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.650469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:02.650469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:02.650469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:02.650469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:02.650469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:02.650469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.650469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:02.650469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:02.650469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.650469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:02.650469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:02.650469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.650469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:02.650469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:02.650469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.650469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:02.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:02.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:02.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:02.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:02.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:02.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:02.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:02.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:02.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:02.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:02.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:02.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:02.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:02.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:02.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:02.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:02.650761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:02.651037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:02.651037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:02.651037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.651037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:02.656190 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:02.656487 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:02.656613 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:02.656699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:02.656966 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:02.657184 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:02.657344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.658025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:02.658025 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:02.658025 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:02.658025 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:02.658025 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:02.658025 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:02.658025 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.658025 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:02.658263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.658263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:02.658263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:02.658263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:02.658263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.658263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:02.658263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:02.658263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.658263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:02.658263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:02.658263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:02.658263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.658488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:02.658488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:02.658488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:02.658488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:02.658488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:02.658488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:02.658488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:02.658488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:02.658488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:02.658488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:02.658488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:02.658488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:02.658488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:02.658737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.658737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:02.658975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.658975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:02.658975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:02.658975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:02.658975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:02.658975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:02.658975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.658975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:02.658975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:02.658975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.658975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:02.658975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:02.658975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.658975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:02.658975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:02.658975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:02.659276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:02.659276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:02.659276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.659276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:02.659276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:02.659276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:02.659276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:02.659276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:02.659276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:02.659276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:02.659276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:02.659276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:02.659276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:02.659276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:02.659276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:02.659276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:02.659276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:02.659276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:02.659276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:02.659693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:02.659693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:02.659693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:03.835573 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:03.836113 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:03.836113 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:03.837141 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:03.870684 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:03.871129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:03.871677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:03.871677 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:03.871677 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:03.871677 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:03.871677 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:03.871677 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:03.871677 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:03.871677 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:03.872871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:03.872871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:03.872871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:03.872871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:03.872871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:03.872871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:03.872871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:03.872871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:03.872871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:03.872871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:03.872871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:03.872871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:03.873369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:03.873369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:03.873369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:03.873369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:03.873369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:03.873369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:03.873369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:03.873369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:03.873369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:03.873369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:03.873369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:03.873369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:03.873369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:03.873819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:03.873819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:03.873819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:03.873819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:03.873819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:03.873819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:03.873819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:03.873819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:03.873819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:03.873819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:03.873819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:03.873819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:03.873819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:03.873819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:03.873819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:03.874354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:03.874354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:03.874354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:03.874354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:03.874354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:03.874354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:03.874354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:03.874354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:03.874354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:03.874354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:03.874354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:03.874354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:03.874354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:03.874354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:03.874354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:03.874354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:03.874354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:03.874354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:03.874354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:03.874967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:03.874967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:03.874967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:03.874967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:03.874967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:03.874967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:07.634677 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:07.635076 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:07.635169 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:07.635939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:07.636187 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:07.636384 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:07.636505 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:07.636613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.637065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:07.637065 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:07.637065 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:07.637065 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:07.637065 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:07.637065 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:07.637065 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.637065 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:07.637932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.637932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:07.637932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:07.637932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:07.637932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.637932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:07.637932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:07.637932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.637932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:07.637932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:07.637932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:07.637932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.638231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:07.638231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:07.638231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:07.638231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:07.638231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:07.638231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:07.638231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:07.638231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:07.638231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:07.638231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:07.638231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:07.638231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:07.638231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:07.638621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.638621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:07.638818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.638818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:07.638818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:07.638818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:07.638818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:07.638818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:07.638818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.638818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:07.638818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:07.638818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.638818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:07.638818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:07.638818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.638818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:07.638818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:07.638818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.639102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:07.639102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:07.639102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.639102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:07.639102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:07.639102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:07.639102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.639102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:07.639102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:07.639102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:07.639102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:07.639102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:07.639102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:07.639102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:07.639102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:07.639102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:07.639102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:07.639102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:07.639102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:07.639416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:07.639416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.639416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:07.639785 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:07.640040 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:07.640165 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:07.640911 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:45:07.640911 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:07.640911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.641433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:07.641433 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:07.641433 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:07.641433 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:07.641433 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:07.641433 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:07.641433 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.641433 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:07.642058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.642058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:07.642058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:07.642058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:07.642058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.642058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:07.642058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:07.642058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:07.642058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.642058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:07.642058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:07.642058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:07.642058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:07.642058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:07.642058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:07.642658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:07.642658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:07.642658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:07.642658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:07.642658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:07.642658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:07.642658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:07.642658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.642658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:07.642658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.642658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:07.642658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:07.642658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:07.642658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.642960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:07.642960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:07.642960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:45:07.642960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:07.642960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:07.642960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.642960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:07.642960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:07.642960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:07.642960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.642960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:07.642960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:07.642960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:07.643199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:07.643199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:07.643199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:07.643199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:07.643199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:07.643199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:07.643199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:07.643199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:07.643199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:07.643199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:07.643199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.643199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:07.643416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.643416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:07.643416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:07.643416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:07.643416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:07.643416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:07.643416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.643416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:07.643416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:07.643416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.643416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:07.643416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:07.643416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.643416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:07.643416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:07.643416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.643712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:07.643712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:07.643712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:45:07.643712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:07.643712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:07.643712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.643712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:07.643712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:07.643712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:07.643712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.643712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:07.643712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:07.643712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:07.643712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:07.643712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:07.643712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:07.643712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:07.643712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:07.643993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:07.643993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:07.643993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:07.643993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:07.643993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:07.643993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.643993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:07.646693 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:07.647017 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:07.647127 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:07.647237 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:07.647579 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:07.647758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.648301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:07.648301 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:07.648301 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:07.648301 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:07.648301 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:07.648301 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:07.648301 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.648301 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:07.648907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.648907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:07.648907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:07.648907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:07.648907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.648907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:07.648907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:07.648907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:07.648907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.648907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:07.648907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:07.648907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:07.648907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:07.648907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:07.648907 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:07.649306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:07.649306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:07.649306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:07.649306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:07.649306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:07.649306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:07.649306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:07.649306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.649306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:07.649750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.649750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:07.649750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:07.649750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:07.649750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.649750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:07.649750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:07.649750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:07.649750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:07.649750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.649750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:07.650475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:07.650475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:07.650475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.650475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:07.650475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:07.650475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:07.650475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:07.650475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:07.650475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:07.650475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:07.650475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:07.650475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:07.650475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:07.650475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:07.650984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:07.650984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:07.650984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.650984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:07.650984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.650984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:07.650984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:07.650984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:07.650984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:07.650984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:07.650984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.650984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:07.650984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:07.650984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.651425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:07.651425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:07.651425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.651425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:07.651425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:07.651425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.651425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:07.651425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:07.651425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:07.651425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:07.651425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.651425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:07.651425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:07.651425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:07.651425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.651425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:07.651425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:07.651425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:07.651425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:07.651760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:07.651760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:07.651760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:07.651760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:07.651760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:07.651760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:07.651760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:07.651760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:07.651760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:07.651760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.651760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:07.656542 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:07.657029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:07.657126 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:07.657336 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:07.657705 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:07.657947 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:07.658166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.658739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:07.658739 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:07.658739 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:07.658739 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:07.658739 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:07.658739 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:07.658739 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.658739 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:07.659659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.659659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:07.659659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:07.659659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:07.659659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.659659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:07.659659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:07.659659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.659659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:07.659659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:07.659659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:07.659659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.660023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:07.660023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:07.660023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:07.660023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:07.660023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:07.660023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:07.660023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:07.660023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:07.660023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:07.660023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:07.660023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:07.660023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:07.660023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:07.660362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.660362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:07.660900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.660900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:07.660900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:07.660900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:07.660900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:07.660900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:07.660900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.660900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:07.660900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:07.660900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.660900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:07.660900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:07.660900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.660900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:07.660900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:07.660900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:07.661241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:07.661241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:07.661241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.661241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:07.661241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:07.661241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:07.661241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:07.661241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:07.661241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:07.661241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:07.661241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:07.661241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:07.661241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:07.661241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:07.661241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:07.661241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:07.661241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:07.661241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:07.661241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:07.661665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:07.661665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:07.661665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:08.836030 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:08.836544 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:08.836703 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:08.837382 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:08.864803 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:08.865033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:08.865636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:08.865636 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:08.865636 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:08.865636 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:08.865636 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:08.865636 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:08.865636 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:08.865636 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:08.866403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:08.866403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:08.866403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:08.866403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:08.866403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:08.866403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:08.866403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:08.866403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:08.866403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:08.866403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:08.866403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:08.866403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:08.866794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:08.866794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:08.866794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:08.866794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:08.866794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:08.866794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:08.866794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:08.866794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:08.866794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:08.866794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:08.866794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:08.866794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:08.866794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:08.867128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:08.867128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:08.867128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:08.867128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:08.867128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:08.867128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:08.867128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:08.867128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:08.867128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:08.867128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:08.867128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:08.867128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:08.867128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:08.867128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:08.867128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:08.867511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:08.867511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:08.867511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:08.867511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:08.867511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:08.867511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:08.867511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:08.867511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:08.867511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:08.867511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:08.867511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:08.867511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:08.867511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:08.867511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:08.867511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:08.867511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:08.867511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:08.867511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:08.867511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:08.867978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:08.867978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:08.867978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:08.867978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:08.867978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:08.867978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:12.635110 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:12.635620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:12.635653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:12.637034 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:12.637034 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:12.637034 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:12.637034 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:12.637034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.637457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:12.637457 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:12.637457 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:12.637457 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:12.637457 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:12.637457 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:12.637457 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.637457 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:12.638262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.638262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:12.638262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:12.638262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:12.638262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.638262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:12.638262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:12.638262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.638262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:12.638262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:12.638262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:12.638262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.638508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:12.638508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:12.638508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:12.638508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:12.638508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:12.638508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:12.638508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:12.638508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:12.638508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:12.638508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:12.638508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:12.638508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:12.638508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:12.638712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.638712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:12.639264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.639264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:12.639264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:12.639264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:12.639264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:12.639264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:12.639264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.639264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:12.639264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:12.639264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.639264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:12.639264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:12.639264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.639264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:12.639264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:12.639264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.639692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:12.639692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:12.639692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.639692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:12.639692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:12.639692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:12.639692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.639692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:12.639692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:12.639692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:12.639692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:12.639692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:12.639692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:12.639692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:12.639692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:12.639692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:12.639692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:12.639692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:12.639692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:12.640006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:12.640006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.640006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:12.640059 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:12.640384 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:12.640384 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:12.640468 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:45:12.640752 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:12.640916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.641268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:12.641268 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:12.641268 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:12.641268 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:12.641268 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:12.641268 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:12.641268 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.641268 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:12.641819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.641819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:12.641819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:12.641819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:12.641819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.641819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:12.641819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:12.641819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:12.641819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.641819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:12.641819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:12.641819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:12.641819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:12.641819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:12.641819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:12.642318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:12.642318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:12.642318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:12.642318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:12.642318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:12.642318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:12.642318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:12.642318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.642318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:12.642577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.642577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:12.642577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:12.642577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:12.642577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.642577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:12.642577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:12.642577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:45:12.642577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:12.642577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:12.642577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.642577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:12.642951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:12.642951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:12.642951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.642951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:12.642951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:12.642951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:12.642951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:12.642951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:12.642951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:12.642951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:12.642951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:12.642951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:12.642951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:12.642951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:12.643323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:12.643323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:12.643323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.643323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:12.643323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.643323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:12.643323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:12.643323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:12.643323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:12.643323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:12.643323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.643323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:12.643323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:12.643323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.643557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:12.643557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:12.643557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.643557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:12.643557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:12.643557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.643557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:12.643557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:12.643557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:45:12.643557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:12.643557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:12.643557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.643557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:12.643557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:12.643557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:12.643557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.643557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:12.643557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:12.643557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:12.643843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:12.643843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:12.643843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:12.643843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:12.643843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:12.643843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:12.643843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:12.643843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:12.643843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:12.643843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:12.643843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.643843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:12.647081 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:12.647358 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:12.647439 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:12.647545 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:12.647842 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:12.648023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.648342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:12.648342 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:12.648342 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:12.648342 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:12.648342 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:12.648342 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:12.648342 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.648342 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:12.648847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.648847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:12.648847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:12.648847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:12.648847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.648847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:12.648847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:12.648847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:12.648847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.648847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:12.648847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:12.648847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:12.648847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:12.648847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:12.648847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:12.649113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:12.649113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:12.649113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:12.649113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:12.649113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:12.649113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:12.649113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:12.649113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.649113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:12.649515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.649515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:12.649515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:12.649515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:12.649515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.649515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:12.649515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:12.649515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:12.649515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:12.649515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.649515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:12.649991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:12.649991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:12.649991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.649991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:12.649991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:12.649991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:12.649991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:12.649991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:12.649991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:12.649991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:12.649991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:12.649991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:12.649991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:12.649991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:12.650461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:12.650461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:12.650461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.650461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:12.650461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.650461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:12.650461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:12.650461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:12.650461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:12.650461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:12.650461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.650461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:12.650461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:12.650461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.650802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:12.650802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:12.650802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.650802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:12.650802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:12.650802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.650802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:12.650802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:12.650802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:12.650802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:12.650802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.650802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:12.650802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:12.650802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:12.650802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.650802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:12.650802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:12.650802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:12.650802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:12.651123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:12.651123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:12.651123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:12.651123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:12.651123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:12.651123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:12.651123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:12.651123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:12.651123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:12.651123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.651123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:12.656972 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:12.657264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:12.657370 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:12.657506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:12.657786 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:12.657968 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:12.658105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.658504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:12.658504 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:12.658504 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:12.658504 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:12.658504 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:12.658504 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:12.658504 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.658504 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:12.659196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.659196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:12.659196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:12.659196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:12.659196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.659196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:12.659196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:12.659196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.659196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:12.659196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:12.659196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:12.659196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.659622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:12.659622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:12.659622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:12.659622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:12.659622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:12.659622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:12.659622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:12.659622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:12.659622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:12.659622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:12.659622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:12.659622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:12.659622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:12.659953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.659953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:12.660227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.660227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:12.660227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:12.660227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:12.660227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:12.660227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:12.660227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.660227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:12.660227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:12.660227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.660227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:12.660227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:12.660227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.660227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:12.660227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:12.660227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:12.660717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:12.660717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:12.660717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.660717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:12.660717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:12.660717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:12.660717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:12.660717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:12.660717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:12.660717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:12.660717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:12.660717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:12.660717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:12.660717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:12.660717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:12.660717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:12.660717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:12.660717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:12.660717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:12.661216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:12.661216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:12.661216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:13.836735 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:13.837331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:13.837331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:13.837818 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:13.868538 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:13.868807 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:13.869305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:13.869305 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:13.869305 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:13.869305 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:13.869305 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:13.869305 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:13.869305 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:13.869305 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:13.870086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:13.870086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:13.870086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:13.870086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:13.870086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:13.870086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:13.870086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:13.870086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:13.870086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:13.870086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:13.870086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:13.870086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:13.870527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:13.870527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:13.870527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:13.870527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:13.870527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:13.870527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:13.870527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:13.870527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:13.870527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:13.870527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:13.870527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:13.870527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:13.870527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:13.870941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:13.870941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:13.871204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:13.871204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:13.871204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:13.871204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:13.871204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:13.871204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:13.871204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:13.871204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:13.871204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:13.871204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:13.871204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:13.871204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:13.871204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:13.871204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:13.871204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:13.871204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:13.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:13.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:13.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:13.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:13.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:13.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:13.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:13.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:13.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:13.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:13.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:13.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:13.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:13.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:13.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:13.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:13.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:13.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:13.871540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:13.871857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:13.871857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:13.871857 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:17.635652 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:17.636324 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:17.636324 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:17.637119 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:17.637413 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:17.637703 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:17.637811 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:17.637897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.638538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:17.638538 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:17.638538 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:17.638538 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:17.638538 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:17.638538 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:17.638538 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.638538 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:17.640338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.640338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:17.640338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:17.640338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:17.640338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.640338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:17.640338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:17.640338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.640338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:17.640338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:17.640338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:17.640338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.640809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:17.640809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:17.640809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:17.640809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:17.640809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:17.640809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:17.640809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:17.640809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:17.640809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:17.640809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:17.640809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:17.640809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:17.640809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:17.641244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.641244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:17.641244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.641244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:17.641244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:17.641244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:17.641244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:17.641244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:17.641244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.641244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:17.641244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:17.641244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.641244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:17.641244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:17.641244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:17.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:17.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:17.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:17.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:17.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:17.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:17.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:17.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:17.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:17.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:17.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:17.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:17.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:17.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:17.641780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:17.642294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:17.642294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:17.642294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:17.642294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:17.642294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.642294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:17.642294 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:17.642294 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:17.642294 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:17.642294 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:45:17.642623 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:17.642623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.642703 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:17.642703 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:17.642703 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:17.642703 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:17.642703 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:17.642703 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:17.642703 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.642703 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:17.643711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.643711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:17.643711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:17.643711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:17.643711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.643711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:17.643711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:17.643711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:17.643711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.643711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:17.643711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:17.643711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:17.643711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:17.643711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:17.643711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:17.645189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:17.645189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:17.645189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:17.645189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:17.645189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:17.645189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:17.645189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:17.645189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.645189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:17.645189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.645189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:17.645189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:17.645189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:17.645189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.646092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:17.646092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:17.646092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:45:17.646092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:17.646092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:17.646092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.646092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:17.646092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:17.646092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:17.646092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.646092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:17.646092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:17.646092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:17.647236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:17.647236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:17.647236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:17.647236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:17.647236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:17.647236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:17.647236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:17.647236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:17.647236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:17.647236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:17.647236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.647236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:17.648099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.648099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:17.648099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:17.648099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:17.648099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:17.648099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:17.648099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.648099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:17.648099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:17.648099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.648099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:17.648099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:17.648099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.648099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:17.648099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:17.648099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:45:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:17.650091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:17.650091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:17.650091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:17.650091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:17.650091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:17.650091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.650091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:17.650091 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:17.650091 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:17.650091 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:17.650746 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:17.650746 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:17.650746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.650746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:17.650746 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:17.650746 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:17.650746 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:17.650746 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:17.650746 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:17.650746 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.650746 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:17.650746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.651307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:17.651307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:17.651307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:17.651307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.651307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:17.651307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:17.651307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:17.651307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.651307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:17.651307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:17.651307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:17.651307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:17.651307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:17.651307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:17.651307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:17.651307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:17.651307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:17.651714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:17.651714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:17.651714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:17.651714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:17.651714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.651714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:17.651714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.651714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:17.651714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:17.651714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:17.651714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.651714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:17.651714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:17.652050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:17.652050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:17.652050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.652050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:17.652050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:17.652050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:17.652050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.652050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:17.652050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:17.652050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:17.652050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:17.652050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:17.652050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:17.652419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:17.652419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:17.652419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:17.652419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:17.652419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:17.652419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:17.652419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:17.652419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.652419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:17.652419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.652419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:17.652419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:17.652670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:17.652670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:17.652670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:17.652670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.652670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:17.652670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:17.652670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.652670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:17.652670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:17.652670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.652670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:17.652670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:17.652670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.652670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:17.652670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:17.652670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:17.652670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:17.652670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.652670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:17.653180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:17.653180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:17.653180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.653180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:17.653180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:17.653180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:17.653180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:17.653180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:17.653180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:17.653180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:17.653180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:17.653180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:17.653180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:17.653180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:17.653180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:17.653180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:17.653180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.653180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:17.657380 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:17.657723 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:17.657830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:17.657970 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:17.658303 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:17.658547 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:17.658759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.659406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:17.659406 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:17.659406 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:17.659406 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:17.659406 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:17.659406 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:17.659406 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.659406 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:17.660201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.660201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:17.660201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:17.660201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:17.660201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.660201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:17.660201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:17.660201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.660201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:17.660201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:17.660201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:17.660201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.660741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:17.660741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:17.660741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:17.660741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:17.660741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:17.660741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:17.660741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:17.660741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:17.660741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:17.660741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:17.660741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:17.660741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:17.660741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:17.661181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.661181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:17.661181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.661181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:17.661181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:17.661181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:17.661181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:17.661181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:17.661181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.661181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:17.661181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:17.661181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.661181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:17.661181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:17.661181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.661562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:17.661562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:17.661562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:17.661562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:17.661562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:17.661562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.661562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:17.661562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:17.661562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:17.661562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:17.661562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:17.661562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:17.661562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:17.661562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:17.661562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:17.661562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:17.661562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:17.661562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:17.661562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:17.661936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:17.661936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:17.661936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:17.661936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:17.661936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:17.661936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:18.837152 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:18.837741 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:18.837741 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:18.838482 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:18.870241 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:18.870422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:18.870885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:18.870885 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:18.870885 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:18.870885 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:18.870885 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:18.870885 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:18.870885 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:18.870885 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:18.871892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:18.871892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:18.871892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:18.871892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:18.871892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:18.871892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:18.871892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:18.871892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:18.871892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:18.871892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:18.871892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:18.871892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:18.872277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:18.872277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:18.872277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:18.872277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:18.872277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:18.872277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:18.872277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:18.872277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:18.872277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:18.872277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:18.872277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:18.872277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:18.872277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:18.872622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:18.872622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:18.872919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:18.872919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:18.872919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:18.872919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:18.872919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:18.872919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:18.872919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:18.872919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:18.872919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:18.872919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:18.872919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:18.872919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:18.872919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:18.872919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:18.872919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:18.872919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:18.873363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:18.873363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:18.873363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:18.873363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:18.873363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:18.873363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:18.873363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:18.873363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:18.873363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:18.873363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:18.873363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:18.873363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:18.873363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:18.873363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:18.873363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:18.873363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:18.873363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:18.873363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:18.873363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:18.873861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:18.873861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:18.873861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:22.636063 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:22.637121 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:22.637237 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:22.638105 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:22.638271 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:22.638388 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:22.638487 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:22.638645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.639055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:22.639055 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:22.639055 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:22.639055 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:22.639055 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:22.639055 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:22.639055 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.639055 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:22.639746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.639746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:22.639746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:22.639746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:22.639746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.639746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:22.639746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:22.639746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.639746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:22.639746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:22.639746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:22.639746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.639963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:22.639963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:22.639963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:22.639963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:22.639963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:22.639963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:22.639963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:22.639963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:22.639963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:22.639963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:22.639963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:22.639963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:22.639963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:22.640158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.640158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:22.640416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.640416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:22.640416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:22.640416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:22.640416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:22.640416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:22.640416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.640416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:22.640416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:22.640416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.640416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:22.640416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:22.640416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.640416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:22.640416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:22.640416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.640811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:22.640811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:22.640811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.640811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:22.640811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:22.640811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:22.640811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.640811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:22.640811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:22.640811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:22.640811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:22.640811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:22.640811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:22.640811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:22.640811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:22.640811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:22.640811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:22.640811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:22.640811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:22.641117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:22.641117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.641117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:22.641424 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:22.642057 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:22.642201 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:22.642405 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:45:22.642700 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:22.642844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.643236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:22.643236 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:22.643236 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:22.643236 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:22.643236 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:22.643236 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:22.643236 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.643236 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:22.644479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.644479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:22.644479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:22.644479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:22.644479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.644479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:22.644479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:22.644479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:22.644479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.644479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:22.644479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:22.644479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:22.644479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:22.644479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:22.644479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:22.644821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:22.644821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:22.644821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:22.644821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:22.644821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:22.644821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:22.644821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:22.644821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.644821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:22.645919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.645919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:22.645919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:22.645919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:22.645919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.645919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:22.645919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:22.645919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:45:22.645919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:22.645919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:22.645919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.645919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:22.646405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:22.646405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:22.646405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.646405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:22.646405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:22.646405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:22.646405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:22.646405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:22.646405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:22.646405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:22.646405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:22.646405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:22.646405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:22.646405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:22.646804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:22.646804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:22.646804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.646804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:22.647049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.647049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:22.647049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:22.647049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:22.647049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:22.647049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:22.647049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.647049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:22.647049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:22.647049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.647049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:22.647049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:22.647049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.647049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:22.647049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:22.647049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.647379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:22.647379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:22.647379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:45:22.647379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:22.647379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:22.647379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.647379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:22.647379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:22.647379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:22.647379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.647379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:22.647379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:22.647379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:22.647379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:22.647379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:22.647379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:22.647379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:22.647379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:22.647801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:22.647801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:22.647801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:22.647801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:22.647801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:22.647801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.647801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:22.647801 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:22.648112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:22.648207 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:22.648350 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:22.648683 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:22.649236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.649467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:22.649467 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:22.649467 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:22.649467 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:22.649467 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:22.649467 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:22.649467 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.649467 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:22.650875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.650875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:22.650875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:22.650875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:22.650875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.650875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:22.650875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:22.650875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:22.650875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.650875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:22.650875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:22.650875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:22.650875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:22.650875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:22.650875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:22.651111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:22.651111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:22.651111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:22.651111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:22.651111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:22.651111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:22.651111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:22.651111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.651111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:22.651111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.651111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:22.651111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:22.651111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:22.651111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.651352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:22.651352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:22.651352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:22.651352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:22.651352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.651352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:22.651352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:22.651352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:22.651352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.651352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:22.651352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:22.651352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:22.651352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:22.651563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:22.651563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:22.651563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:22.651563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:22.651563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:22.651563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:22.651563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:22.651563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:22.651563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:22.651563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.651563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:22.651732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.651732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:22.651732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:22.651732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:22.651732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:22.651732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:22.651732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.651732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:22.651732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:22.651732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.651732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:22.651732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:22.651732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.651732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:22.651732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:22.651732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.651732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:22.652028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:22.652028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:22.652028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:22.652028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.652028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:22.652028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:22.652028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:22.652028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.652028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:22.652028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:22.652028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:22.652028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:22.652028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:22.652028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:22.652028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:22.652028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:22.652028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:22.652028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:22.652028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:22.652343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:22.652343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:22.652343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.652343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:22.657784 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:22.658208 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:22.658333 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:22.658761 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:22.659074 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:22.659332 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:22.659526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.660085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:22.660085 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:22.660085 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:22.660085 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:22.660085 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:22.660085 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:22.660085 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.660085 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:22.660911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.660911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:22.660911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:22.660911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:22.660911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.660911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:22.660911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:22.660911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.660911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:22.660911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:22.660911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:22.660911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.661300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:22.661300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:22.661300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:22.661300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:22.661300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:22.661300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:22.661300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:22.661300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:22.661300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:22.661300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:22.661300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:22.661300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:22.661300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:22.661611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.661611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:22.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:22.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:22.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:22.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:22.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:22.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:22.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:22.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:22.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:22.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:22.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:22.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:22.662163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:22.662163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:22.662163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.662163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:22.662163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:22.662163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:22.662163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:22.662163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:22.662163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:22.662163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:22.662163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:22.662163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:22.662163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:22.662163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:22.662163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:22.662163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:22.662163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:22.662163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:22.662163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:22.662707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:22.662707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:22.662707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:23.837791 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:23.838804 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:23.839014 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:23.840025 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:23.877430 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:23.877714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:23.878154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:23.878154 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:23.878154 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:23.878154 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:23.878154 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:23.878154 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:23.878154 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:23.878154 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:23.879041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:23.879041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:23.879041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:23.879041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:23.879041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:23.879041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:23.879041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:23.879041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:23.879041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:23.879041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:23.879041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:23.879041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:23.879881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:23.879881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:23.879881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:23.879881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:23.879881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:23.879881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:23.879881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:23.879881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:23.879881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:23.879881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:23.879881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:23.879881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:23.879881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:23.880579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:23.880579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:23.880579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:23.880579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:23.880579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:23.880579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:23.880579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:23.880579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:23.880579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:23.880579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:23.880579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:23.880579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:23.880579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:23.880579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:23.880579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:23.881159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:23.881159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:23.881159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:23.881159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:23.881159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:23.881159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:23.881159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:23.881159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:23.881159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:23.881159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:23.881159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:23.881159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:23.881159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:23.881159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:23.881159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:23.881159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:23.881159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:23.881159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:23.881159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:23.881847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:23.881847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:23.881847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:23.881847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:23.881847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:23.881847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:27.637234 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:27.637940 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:27.637940 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:27.639689 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:27.639689 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:27.639689 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:27.639689 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:27.639689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.639961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:27.639961 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:27.639961 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:27.639961 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:27.639961 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:27.639961 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:27.639961 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.639961 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:27.640756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.640756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:27.640756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:27.640756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:27.640756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.640756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:27.640756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:27.640756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.640756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:27.640756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:27.640756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:27.640756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.641245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:27.641245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:27.641245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:27.641245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:27.641245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:27.641245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:27.641245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:27.641245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:27.641245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:27.641245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:27.641245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:27.641245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:27.641245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:27.641751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.641751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:27.641841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.641841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:27.641841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:27.641841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:27.641841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:27.641841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:27.641841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.641841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:27.641841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:27.641841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.641841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:27.641841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:27.641841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.641841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:27.641841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:27.641841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.642382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:27.642382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:27.642382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.642382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:27.642382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:27.642382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:27.642382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.642382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:27.642382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:27.642382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:27.642382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:27.642382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:27.642382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:27.642382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:27.642382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:27.642382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:27.642382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:27.642382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:27.642382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:27.643042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:27.643042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.643042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:27.643042 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:27.643042 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:27.643042 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:27.643042 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:45:27.643642 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:27.643928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.644535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:27.644535 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:27.644535 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:27.644535 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:27.644535 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:27.644535 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:27.644535 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.644535 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:27.645310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.645310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:27.645310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:27.645310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:27.645310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.645310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:27.645310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:27.645310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:27.645310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.645310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:27.645310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:27.645310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:27.645310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:27.645310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:27.645310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:27.645875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:27.645875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:27.645875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:27.645875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:27.645875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:27.645875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:27.645875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:27.645875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.645875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:27.646195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.646195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:27.646195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:27.646195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:27.646195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.646195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:27.646195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:27.646195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:45:27.646195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:27.646195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:27.646195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.646195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:27.646617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:27.646617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:27.646617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.646617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:27.646617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:27.646617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:27.646617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:27.646617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:27.646617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:27.646617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:27.646617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:27.646617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:27.646617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:27.646617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:27.647025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:27.647025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:27.647025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.647025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:27.647263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.647263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:27.647263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:27.647263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:27.647263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:27.647263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:27.647263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.647263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:27.647263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:27.647263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.647263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:27.647263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:27.647263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.647263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:27.647263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:27.647263 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.647804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:27.647804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:27.647804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:45:27.647804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:27.647804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:27.647804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.647804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:27.647804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:27.647804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:27.647804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.647804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:27.647804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:27.647804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:27.647804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:27.647804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:27.647804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:27.647804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:27.647804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:27.648347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:27.648347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:27.648347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:27.648347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:27.648347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:27.648347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.648347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:27.648347 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:27.648613 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:27.648659 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:27.648771 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:27.649386 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:27.649386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.649840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:27.649840 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:27.649840 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:27.649840 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:27.649840 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:27.649840 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:27.649840 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.649840 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:27.650601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.650601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:27.650601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:27.650601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:27.650601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.650601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:27.650601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:27.650601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:27.650601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.650601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:27.650601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:27.650601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:27.650601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:27.650601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:27.650601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:27.650920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:27.650920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:27.650920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:27.650920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:27.650920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:27.650920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:27.650920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:27.650920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.650920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:27.651579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.651579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:27.651579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:27.651579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:27.651579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.651579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:27.651579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:27.651579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:27.651579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:27.651579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.651579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:27.651824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:27.651824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:27.651824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.651824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:27.651824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:27.651824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:27.651824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:27.651824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:27.651824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:27.651824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:27.651824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:27.651824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:27.651824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:27.651824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:27.652103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:27.652103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:27.652103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.652103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:27.652751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.652751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:27.652751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:27.652751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:27.652751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:27.652751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:27.652751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.652751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:27.652751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:27.652751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.652751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:27.652751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:27.652751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.652751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:27.652751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:27.652751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.652751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:27.653058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:27.653058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:27.653058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:27.653058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.653058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:27.653058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:27.653058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:27.653058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.653058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:27.653058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:27.653058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:27.653058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:27.653058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:27.653058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:27.653058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:27.653058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:27.653058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:27.653058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:27.653058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:27.653381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:27.653381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:27.653381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.653381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:27.658197 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:27.658590 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:27.658723 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:27.658913 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:27.659246 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:27.659489 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:27.659674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.660174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:27.660174 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:27.660174 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:27.660174 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:27.660174 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:27.660174 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:27.660174 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.660174 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:27.661034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.661034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:27.661034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:27.661034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:27.661034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.661034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:27.661034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:27.661034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.661034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:27.661034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:27.661034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:27.661034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.661298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:27.661298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:27.661298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:27.661298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:27.661298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:27.661298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:27.661298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:27.661298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:27.661298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:27.661298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:27.661298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:27.661298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:27.661298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:27.661575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.661575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:27.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:27.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:27.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:27.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:27.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:27.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:27.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:27.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:27.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:27.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:27.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:27.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:27.662494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:27.662494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:27.662494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.662494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:27.662494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:27.662494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:27.662494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:27.662494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:27.662494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:27.662494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:27.662494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:27.662494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:27.662494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:27.662494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:27.662494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:27.662494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:27.662494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:27.662494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:27.662494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:27.662810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:27.662810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:27.662810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:28.838754 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:28.839347 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:28.839421 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:28.840423 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:28.876760 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:28.877064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:28.877763 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:28.877763 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:28.877763 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:28.877763 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:28.877763 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:28.877763 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:28.877763 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:28.877763 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:28.878860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:28.878860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:28.878860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:28.878860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:28.878860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:28.878860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:28.878860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:28.878860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:28.878860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:28.878860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:28.878860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:28.878860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:28.879245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:28.879245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:28.879245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:28.879245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:28.879245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:28.879245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:28.879245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:28.879245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:28.879245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:28.879245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:28.879245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:28.879245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:28.879245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:28.879630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:28.879630 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:28.880062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:28.880062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:28.880062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:28.880062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:28.880062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:28.880062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:28.880062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:28.880062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:28.880062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:28.880062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:28.880062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:28.880062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:28.880062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:28.880062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:28.880062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:28.880062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:28.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:28.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:28.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:28.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:28.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:28.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:28.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:28.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:28.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:28.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:28.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:28.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:28.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:28.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:28.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:28.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:28.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:28.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:28.880533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:28.881073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:28.881073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:28.881073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:32.637716 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:32.638186 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:32.638186 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:32.638884 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:32.639055 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:32.639215 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:32.639398 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-14c7d95c-12a9-4510-9230-8fd6409f0621, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:32.639542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.640097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:32.640097 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:32.640097 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:32.640097 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:32.640097 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:32.640097 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:32.640097 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.640097 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:32.640764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.640764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:32.640764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:32.640764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:32.640764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.640764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:32.640764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:32.640764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.640764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:32.640764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:32.640764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:32.640764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.641009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:32.641009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:32.641009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:32.641009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:32.641009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:32.641009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:32.641009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:32.641009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:32.641009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:32.641009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:32.641009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:32.641009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:32.641009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:32.641226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.641226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:32.641515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.641515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:32.641515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:32.641515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:32.641515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:32.641515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:32.641515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.641515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:32.641515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:32.641515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.641515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:32.641515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:32.641515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.641515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:32.641515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:32.641515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.641924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:32.641924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:32.641924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.641924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:32.641924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:32.641924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:32.641924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.641924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:32.641924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:32.641924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:32.641924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:32.641924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:32.641924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:32.641924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:32.641924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:32.641924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:32.641924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:32.641924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:32.641924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:32.642301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:32.642301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.642301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:32.642538 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:32.642725 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:32.642809 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:32.642897 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:45:32.643214 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:32.643402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.643779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:32.643779 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:32.643779 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:32.643779 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:32.643779 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:32.643779 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:32.643779 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.643779 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:32.644321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.644321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:32.644321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:32.644321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:32.644321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.644321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:32.644321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:32.644321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:32.644321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.644321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:32.644321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:32.644321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:32.644321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:32.644321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:32.644321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:32.644591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:32.644591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:32.644591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:32.644591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:32.644591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:32.644591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:32.644591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:32.644591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.644591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:32.644872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.644872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:32.644872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:32.644872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:32.644872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.644872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:32.644872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:32.644872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:45:32.644872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:32.644872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:32.644872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.644872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:32.645078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:32.645078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:32.645078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.645078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:32.645078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:32.645078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:32.645078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:32.645078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:32.645078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:32.645078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:32.645078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:32.645078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:32.645078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:32.645078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:32.645314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:32.645314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:32.645314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.645314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:32.645594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.645594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:32.645594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:32.645594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:32.645594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:32.645594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:32.645594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.645594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:32.645594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:32.645594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.645594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:32.645594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:32.645594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.645594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:32.645594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:32.645594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.645903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:32.645903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:32.645903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:45:32.645903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:32.645903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:32.645903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.645903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:32.645903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:32.645903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:32.645903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.645903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:32.645903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:32.645903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:32.645903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:32.645903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:32.645903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:32.645903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:32.645903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:32.646178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:32.646178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:32.646178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:32.646178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:32.646178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:32.646178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.646178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:32.648577 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:32.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:32.648944 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:32.649042 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:32.649366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:32.649553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:32.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:32.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:32.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:32.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:32.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:32.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.649936 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:32.650484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.650484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:32.650484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:32.650484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:32.650484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.650484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:32.650484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:32.650484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:32.650484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.650484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:32.650484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:32.650484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:32.650484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:32.650484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:32.650484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:32.650780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:32.650780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:32.650780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:32.650780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:32.650780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:32.650780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:32.650780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:32.650780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.650780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:32.651088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.651088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:32.651088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:32.651088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:32.651088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.651088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:32.651088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:32.651088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:32.651088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:32.651088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.651088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:32.651291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:32.651291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:32.651291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.651291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:32.651291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:32.651291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:32.651291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:32.651291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:32.651291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:32.651291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:32.651291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:32.651291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:32.651291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:32.651291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:32.651536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:32.651536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:32.651536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.651536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:32.651833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.651833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:32.651833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:32.651833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:32.651833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:32.651833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:32.651833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.651833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:32.651833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:32.651833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.651833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:32.651833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:32.651833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.651833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:32.651833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:32.651833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.651833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:32.652159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:32.652159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:32.652159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:32.652159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.652159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:32.652159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:32.652159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:32.652159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.652159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:32.652159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:32.652159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:32.652159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:32.652159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:32.652159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:32.652159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:32.652159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:32.652159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:32.652159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:32.652159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:32.652464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:32.652464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:32.652464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.652464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:32.658584 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:32.659050 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:32.659237 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:32.659484 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:32.659832 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:32.660028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:32.660178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.660727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:32.660727 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:32.660727 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:32.660727 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:32.660727 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:32.660727 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:32.660727 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.660727 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:32.661725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.661725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:32.661725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:32.661725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:32.661725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.661725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:32.661725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:32.661725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.661725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:32.661725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:32.661725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:32.661725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.661954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:32.661954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:32.661954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:32.661954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:32.661954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:32.661954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:32.661954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:32.661954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:32.661954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:32.661954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:32.661954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:32.661954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:32.661954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:32.662160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.662160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:32.662895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.662895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:32.662895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:32.662895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:32.662895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:32.662895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:32.662895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.662895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:32.662895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:32.662895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.662895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:32.662895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:32.662895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.662895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:32.662895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:32.662895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:32.663221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:32.663221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:32.663221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.663221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:32.663221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:32.663221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:32.663221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:32.663221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:32.663221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:32.663221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:32.663221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:32.663221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:32.663221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:32.663221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:32.663221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:32.663221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:32.663221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:32.663221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:32.663221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:32.663552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:32.663552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:32.663552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:33.839207 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:33.839839 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:33.839839 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:33.840341 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:33.868982 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:33.869100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:33.869683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:33.869683 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:33.869683 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:33.869683 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:33.869683 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:33.869683 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:33.869683 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:33.869683 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:33.870302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:33.870302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:33.870302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:33.870302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:33.870302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:33.870302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:33.870302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:33.870302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:33.870302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:33.870302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:33.870302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:33.870302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:33.871137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:33.871137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:33.871137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:33.871137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:33.871137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:33.871137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:33.871137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:33.871137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:33.871137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:33.871137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:33.871137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:33.871137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:33.871137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:33.871362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:33.871362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:33.871362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:33.871362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:33.871362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:33.871362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:33.871362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:33.871362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:33.871362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:33.871362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:33.871362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:33.871362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:33.871362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:33.871362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:33.871362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:33.871619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:33.871619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:33.871619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:33.871619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:33.871619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:33.871619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:33.871619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:33.871619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:33.871619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:33.871619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:33.871619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:33.871619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:33.871619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:33.871619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:33.871619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:33.871619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:33.871619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:33.871619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:33.871619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:33.871921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:33.871921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:33.871921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:33.871921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:33.871921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:33.871921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:37.638169 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:37.638781 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:37.638781 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:37.639367 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:37.639592 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:37.639740 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:37.639852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.640298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:37.640298 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:37.640298 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:37.640298 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:37.640298 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:37.640298 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:37.640298 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.640298 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:37.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:37.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:37.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:37.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:37.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:37.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:37.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:37.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:37.640977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.641264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:37.641264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:37.641264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:37.641264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:37.641264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:37.641264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:37.641264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:37.641264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:37.641264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:37.641264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:37.641264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:37.641264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:37.641264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:37.641536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.641536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:37.641777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.641777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:37.641777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:37.641777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:37.641777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:37.641777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:37.641777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.641777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:37.641777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:37.641777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.641777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:37.641777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:37.641777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.641777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:37.641777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:37.641777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.642117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:37.642117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:37.642117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.642117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:37.642117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:37.642117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:37.642117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.642117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:37.642117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:37.642117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:37.642117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:37.642117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:37.642117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:37.642117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:37.642117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:37.642117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:37.642117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:37.642117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:37.642117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:37.642475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:37.642475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.642475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:37.642793 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:37.643010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:37.643100 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:37.643188 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:45:37.643454 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:37.643623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.643971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:37.643971 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:37.643971 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:37.643971 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:37.643971 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:37.643971 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:37.643971 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.643971 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:37.644469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.644469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:37.644469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:37.644469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:37.644469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.644469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:37.644469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:37.644469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:37.644469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.644469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:37.644469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:37.644469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:37.644469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:37.644469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:37.644469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:37.644830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:37.644830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:37.644830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:37.644830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:37.644830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:37.644830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:37.644830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:37.644830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.644830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:37.645086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.645086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:37.645086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:37.645086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:37.645086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.645086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:37.645086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:37.645086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:45:37.645086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:37.645086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:37.645086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.645086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:37.645342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:37.645342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:37.645342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.645342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:37.645342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:37.645342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:37.645342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:37.645342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:37.645342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:37.645342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:37.645342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:37.645342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:37.645342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:37.645342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:37.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:37.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:37.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:37.645820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.645820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:37.645820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:37.645820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:37.645820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:37.645820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:37.645820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.645820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:37.645820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:37.645820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.645820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:37.645820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:37.645820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.645820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:37.645820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:37.645820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.646074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:37.646074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:37.646074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:45:37.646074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:37.646074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:37.646074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.646074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:37.646074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:37.646074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:37.646074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.646074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:37.646074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:37.646074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:37.646074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:37.646074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:37.646074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:37.646074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:37.646074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:37.646408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:37.646408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:37.646408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:37.646408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:37.646408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:37.646408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.646408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:37.648857 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:37.649071 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:37.649157 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:37.649267 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:37.649498 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:37.649702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.650040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:37.650040 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:37.650040 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:37.650040 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:37.650040 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:37.650040 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:37.650040 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.650040 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:37.650548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.650548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:37.650548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:37.650548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:37.650548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.650548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:37.650548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:37.650548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:37.650548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.650548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:37.650548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:37.650548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:37.650548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:37.650548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:37.650548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:37.650849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:37.650849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:37.650849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:37.650849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:37.650849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:37.650849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:37.650849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:37.650849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.650849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:37.651208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.651208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:37.651208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:37.651208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:37.651208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.651208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:37.651208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:37.651208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:37.651208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:37.651208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.651208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:37.651445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:37.651445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:37.651445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.651445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:37.651445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:37.651445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:37.651445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:37.651445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:37.651445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:37.651445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:37.651445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:37.651445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:37.651445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:37.651445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:37.651678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:37.651678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:37.651678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.651678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:37.651971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.651971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:37.651971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:37.651971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:37.651971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:37.651971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:37.651971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.651971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:37.651971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:37.651971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.651971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:37.651971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:37.651971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.651971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:37.651971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:37.651971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.651971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:37.652271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:37.652271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:37.652271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:37.652271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.652271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:37.652271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:37.652271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:37.652271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.652271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:37.652271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:37.652271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:37.652271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:37.652271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:37.652271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:37.652271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:37.652271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:37.652271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:37.652271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:37.652271 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:37.652722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:37.652722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:37.652722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.652722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:37.658946 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:37.659199 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:37.659316 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:37.659454 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:37.660311 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:37.660311 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:37.660311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.661281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:37.661281 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:37.661281 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:37.661281 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:37.661281 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:37.661281 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:37.661281 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.661281 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:37.661466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.661466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:37.661466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:37.661466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:37.661466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.661466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:37.661466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:37.661466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.661466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:37.661466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:37.661466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:37.661466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.661709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:37.661709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:37.661709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:37.661709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:37.661709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:37.661709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:37.661709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:37.661709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:37.661709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:37.661709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:37.661709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:37.661709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:37.661709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:37.661914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.661914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:37.662072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.662072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:37.662072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:37.662072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:37.662072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:37.662072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:37.662072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.662072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:37.662072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:37.662072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.662072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:37.662072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:37.662072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.662072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:37.662072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:37.662072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:37.662346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:37.662346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:37.662346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.662346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:37.662346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:37.662346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:37.662346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:37.662346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:37.662346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:37.662346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:37.662346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:37.662346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:37.662346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:37.662346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:37.662346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:37.662346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:37.662346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:37.662346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:37.662346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:37.662710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:37.662710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:37.662710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:38.839687 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:38.840397 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:38.840397 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:38.841179 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:38.879185 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:38.879185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:38.879408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:38.879408 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:38.879408 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:38.879408 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:38.879408 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:38.879408 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:38.879408 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:38.879408 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:38.880180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:38.880180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:38.880180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:38.880180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:38.880180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:38.880180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:38.880180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:38.880180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:38.880180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:38.880180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:38.880180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:38.880180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:38.880618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:38.880618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:38.880618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:38.880618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:38.880618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:38.880618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:38.880618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:38.880618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:38.880618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:38.880618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:38.880618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:38.880618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:38.880618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:38.880956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:38.880956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:38.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:38.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:38.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:38.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:38.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:38.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:38.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:38.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:38.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:38.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:38.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:38.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:38.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:38.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:38.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:38.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:38.881619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:38.881619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:38.881619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:38.881619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:38.881619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:38.881619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:38.881619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:38.881619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:38.881619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:38.881619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:38.881619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:38.881619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:38.881619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:38.881619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:38.881619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:38.881619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:38.881619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:38.881619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:38.881619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:38.882132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:38.882132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:38.882132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:42.638634 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:42.640024 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:42.640024 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:42.640292 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:42.640464 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:42.640611 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:42.640732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.642679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:42.642679 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:42.642679 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:42.642679 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:42.642679 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:42.642679 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:42.642679 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.642679 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:42.642679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.642679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:42.642679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:42.642679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:42.642679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.642679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:42.642679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:42.642996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.642996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:42.642996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:42.642996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:42.642996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.642996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:42.642996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:42.642996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:42.642996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:42.642996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:42.642996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:42.642996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:42.642996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:42.642996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:42.643282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:42.643282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:42.643282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:42.643282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:42.643282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.643282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:42.643282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.643282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:42.643282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:42.643282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:42.643282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:42.643282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:42.643282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.643282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:42.643516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:42.643883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:42.643883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:42.643883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:42.643883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:42.643883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:42.643883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:42.643883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:42.643883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:42.643883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.643883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:42.644066 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:42.644703 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:42.644703 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:42.644703 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:45:42.644891 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:42.645057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.647535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:42.647535 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:42.647535 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:42.647535 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:42.647535 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:42.647535 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:42.647535 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.647535 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:42.647535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.647535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:42.647535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:42.647535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:42.647535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.647535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:42.647535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:42.647535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:42.647535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.647535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:42.647928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:42.647928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:42.647928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:42.647928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:42.647928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:42.647928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:42.647928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:42.647928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:42.647928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:42.647928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:42.647928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:42.647928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:42.647928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.647928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:42.647928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.648205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:42.648205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:42.648205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:42.648205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.648205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:42.648205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:42.648205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:45:42.648205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:42.648205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:42.648205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.648205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:42.648205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:42.648205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:42.648460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.648460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:42.648460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:42.648460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:42.648460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:42.648460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:42.648460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:42.648460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:42.648460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:42.648460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:42.648460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:42.648460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:42.648460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:42.648460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:42.648713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.648713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:42.648713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.648713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:42.648713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:42.648713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:42.648713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:42.648713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:42.648713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.648713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:42.648713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:42.648713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.648713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:42.648713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:42.648713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.648982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:42.648982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:42.648982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.648982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:42.648982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:42.648982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:45:42.648982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:42.648982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:42.648982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.648982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:42.648982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:42.648982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:42.648982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.648982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:42.648982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:42.648982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:42.648982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:42.648982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:42.648982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:42.649323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:42.649323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:42.649323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:42.649323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:42.649323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:42.649323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:42.649323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:42.649323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.649323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:42.649548 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:42.650314 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:42.650314 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:42.650314 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:42.650525 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:42.650645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.652835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:42.652835 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:42.652835 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:42.652835 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:42.652835 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:42.652835 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:42.652835 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.652835 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:42.652835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.652835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:42.652835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:42.652835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:42.652835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.652835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:42.652835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:42.652835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:42.652835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.652835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:42.653369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:42.653369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:42.653369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:42.653369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:42.653369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:42.653369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:42.653369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:42.653369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:42.653369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:42.653369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:42.653369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:42.653369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:42.653369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.653369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:42.653369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.653821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:42.653821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:42.653821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:42.653821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.653821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:42.653821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:42.653821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:42.653821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:42.653821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.653821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:42.653821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:42.653821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:42.653821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.654178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:42.654178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:42.654178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:42.654178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:42.654178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:42.654178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:42.654178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:42.654178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:42.654178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:42.654178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:42.654178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:42.654178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:42.654178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:42.654578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.654578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:42.654578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.654578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:42.654578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:42.654578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:42.654578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:42.654578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:42.654578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.654578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:42.654578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:42.654578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.654578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:42.654578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:42.654578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.655002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:42.655002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:42.655002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.655002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:42.655002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:42.655002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:42.655002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:42.655002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.655002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:42.655002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:42.655002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:42.655002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.655002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:42.655002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:42.655002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:42.655002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:42.655002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:42.655002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:42.655002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:42.655544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:42.655544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:42.655544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:42.655544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:42.655544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:42.655544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:42.655544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.655544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:42.659525 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:42.660182 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:42.660182 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:42.660182 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:42.660869 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:42.660869 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:42.660869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.662684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:42.662684 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:42.662684 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:42.662684 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:42.662684 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:42.662684 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:42.662684 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.662684 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:42.662684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.662684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:42.662684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:42.662684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:42.662684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.662684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:42.662684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:42.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:42.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:42.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:42.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:42.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:42.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:42.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:42.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:42.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:42.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:42.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:42.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:42.663369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:42.663369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:42.663369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:42.663369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:42.663369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.663369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:42.663369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.663369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:42.663369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:42.663369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:42.663369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:42.663369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:42.663369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.663369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:42.663712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:42.664067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:42.664067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:42.664067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:42.664067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:42.664067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:42.664067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:42.664067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:42.664067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:42.664067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:42.664067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:43.840178 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:43.842183 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:43.842183 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:43.842183 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:43.878484 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:43.878767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:43.879306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:43.879306 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:43.879306 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:43.879306 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:43.879306 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:43.879306 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:43.879306 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:43.879306 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:43.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:43.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:43.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:43.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:43.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:43.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:43.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:43.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:43.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:43.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:43.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:43.881234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:43.881517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:43.881517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:43.881517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:43.881517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:43.881517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:43.881517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:43.881517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:43.881517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:43.881517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:43.881517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:43.881517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:43.881517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:43.881517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:43.881757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:43.881757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:43.881757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:43.881757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:43.881757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:43.881757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:43.881757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:43.881757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:43.881757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:43.881757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:43.881757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:43.881757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:43.881757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:43.881757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:43.881757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:43.882005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:43.882005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:43.882005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:43.882005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:43.882005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:43.882005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:43.882005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:43.882005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:43.882005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:43.882005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:43.882005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:43.882005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:43.882005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:43.882005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:43.882005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:43.882005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:43.882005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:43.882005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:43.882005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:43.882297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:43.882297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:43.882297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:43.882297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:43.882297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:43.882297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:47.639616 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:47.640129 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:47.640167 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:47.641320 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:47.641320 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:47.641320 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:47.641320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.641736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:47.641736 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:47.641736 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:47.641736 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:47.641736 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:47.641736 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:47.641736 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.641736 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:47.642712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.642712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:47.642712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:47.642712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:47.642712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.642712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:47.642712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:47.642712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.642712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:47.642712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:47.642712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:47.642712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.642984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:47.642984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:47.642984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:47.642984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:47.642984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:47.642984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:47.642984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:47.642984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:47.642984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:47.642984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:47.642984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:47.642984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:47.642984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:47.643243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.643243 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:47.643863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.643863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:47.643863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:47.643863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:47.643863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:47.643863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:47.643863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.643863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:47.643863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:47.643863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.643863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:47.643863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:47.643863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.643863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:47.643863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:47.643863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.644209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:47.644209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:47.644209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.644209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:47.644209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:47.644209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:47.644209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.644209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:47.644209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:47.644209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:47.644209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:47.644209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:47.644209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:47.644209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:47.644209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:47.644209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:47.644209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:47.644209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:47.644209 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:47.644711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:47.644711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.644711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:47.644711 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:47.644860 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:47.644982 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:47.645114 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:45:47.645436 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:47.645603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.645966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:47.645966 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:47.645966 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:47.645966 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:47.645966 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:47.645966 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:47.645966 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.645966 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:47.646760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.646760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:47.646760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:47.646760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:47.646760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.646760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:47.646760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:47.646760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:47.646760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.646760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:47.646760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:47.646760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:47.646760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:47.646760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:47.646760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:47.647102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:47.647102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:47.647102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:47.647102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:47.647102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:47.647102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:47.647102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:47.647102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.647102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:47.647663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.647663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:47.647663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:47.647663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:47.647663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.647663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:47.647663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:47.647663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:45:47.647663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:47.647663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:47.647663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.647663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:47.647920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:47.647920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:47.647920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.647920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:47.647920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:47.647920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:47.647920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:47.647920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:47.647920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:47.647920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:47.647920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:47.647920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:47.647920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:47.647920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:47.648133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:47.648133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:47.648133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.648133 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:47.648817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.648817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:47.648817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:47.648817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:47.648817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:47.648817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:47.648817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.648817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:47.648817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:47.648817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.648817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:47.648817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:47.648817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.648817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:47.648817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:47.648817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.649097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:47.649097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:47.649097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:45:47.649097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:47.649097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:47.649097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.649097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:47.649097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:47.649097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:47.649097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.649097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:47.649097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:47.649097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:47.649097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:47.649097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:47.649097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:47.649097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:47.649097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:47.649411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:47.649411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:47.649411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:47.649411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:47.649411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:47.649411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.649411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:47.650023 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:47.650322 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:47.650444 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:47.650591 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:47.650888 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:47.651097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.651511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:47.651511 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:47.651511 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:47.651511 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:47.651511 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:47.651511 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:47.651511 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.651511 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:47.652332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.652332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:47.652332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:47.652332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:47.652332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.652332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:47.652332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:47.652332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:47.652332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.652332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:47.652332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:47.652332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:47.652332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:47.652332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:47.652332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:47.652581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:47.652581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:47.652581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:47.652581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:47.652581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:47.652581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:47.652581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:47.652581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.652581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:47.653296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.653296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:47.653296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:47.653296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:47.653296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.653296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:47.653296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:47.653296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:47.653296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:47.653296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.653296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:47.653521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:47.653521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:47.653521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.653521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:47.653521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:47.653521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:47.653521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:47.653521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:47.653521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:47.653521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:47.653521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:47.653521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:47.653521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:47.653521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:47.653815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:47.653815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:47.653815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.653815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:47.654626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.654626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:47.654626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:47.654626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:47.654626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:47.654626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:47.654626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.654626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:47.654626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:47.654626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.654626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:47.654626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:47.654626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.654626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:47.654626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:47.654626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.654626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:47.655041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:47.655041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:47.655041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:47.655041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.655041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:47.655041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:47.655041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:47.655041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.655041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:47.655041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:47.655041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:47.655041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:47.655041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:47.655041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:47.655041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:47.655041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:47.655041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:47.655041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:47.655041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:47.655361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:47.655361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:47.655361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.655361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:47.660000 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:47.660305 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:47.660416 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:47.660581 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:47.660874 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:47.661116 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:47.661309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:47.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:47.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:47.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:47.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:47.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:47.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.661773 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:47.662385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.662385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:47.662385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:47.662385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:47.662385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.662385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:47.662385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:47.662385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.662385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:47.662385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:47.662385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:47.662385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.662653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:47.662653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:47.662653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:47.662653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:47.662653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:47.662653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:47.662653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:47.662653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:47.662653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:47.662653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:47.662653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:47.662653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:47.662653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:47.662885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.662885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:47.663047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.663047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:47.663047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:47.663047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:47.663047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:47.663047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:47.663047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.663047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:47.663047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:47.663047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.663047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:47.663047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:47.663047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.663047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:47.663047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:47.663047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:47.663395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:47.663395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:47.663395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.663395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:47.663395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:47.663395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:47.663395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:47.663395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:47.663395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:47.663395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:47.663395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:47.663395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:47.663395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:47.663395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:47.663395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:47.663395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:47.663395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:47.663395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:47.663395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:47.663835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:47.663835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:47.663835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:48.841152 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:48.841703 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:48.841703 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:48.842428 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:48.872314 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:48.872605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:48.873031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:48.873031 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:48.873031 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:48.873031 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:48.873031 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:48.873031 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:48.873031 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:48.873031 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:48.873707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:48.873707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:48.873707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:48.873707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:48.873707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:48.873707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:48.873707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:48.873707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:48.873707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:48.873707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:48.873707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:48.873707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:48.873997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:48.873997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:48.873997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:48.873997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:48.873997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:48.873997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:48.873997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:48.873997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:48.873997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:48.873997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:48.873997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:48.873997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:48.873997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:48.874197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:48.874197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:48.874536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:48.874536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:48.874536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:48.874536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:48.874536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:48.874536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:48.874536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:48.874536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:48.874536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:48.874536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:48.874536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:48.874536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:48.874536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:48.874536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:48.874536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:48.874536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:48.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:48.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:48.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:48.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:48.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:48.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:48.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:48.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:48.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:48.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:48.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:48.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:48.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:48.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:48.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:48.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:48.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:48.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:48.875395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:48.875835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:48.875835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:48.875835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:52.640040 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:52.641342 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:52.641342 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:52.641686 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:52.641966 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:52.642218 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:52.642461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.642966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:52.642966 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:52.642966 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:52.642966 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:52.642966 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:52.642966 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:52.642966 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.642966 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:52.643926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.643926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:52.643926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:52.643926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:52.643926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.643926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:52.643926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:52.643926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.643926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:52.643926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:52.643926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:52.643926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.644167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:52.644167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:52.644167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:52.644167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:52.644167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:52.644167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:52.644167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:52.644167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:52.644167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:52.644167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:52.644167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:52.644167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:52.644167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:52.644477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.644477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:52.645636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.645636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:52.645636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:52.645636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:52.645636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:52.645636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:52.645636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.645636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:52.645636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:52.645636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.645636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:52.645636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:52.645636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.645636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:52.645636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:52.645636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.646644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:52.646644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:52.646644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.646644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:52.646644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:52.646644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:52.646644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.646644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:52.646644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:52.646644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:52.646644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:52.646644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:52.646644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:52.646644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:52.646644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:52.646644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:52.646644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:52.646644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:52.646644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:52.647193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:52.647193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.647193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:52.647193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:52.647193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:52.647193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:52.647193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:45:52.647193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:52.649613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.649613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:52.649613 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:52.649613 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:52.649613 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:52.649613 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:52.649613 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:52.649613 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.649613 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:52.649613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.649613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:52.649613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:52.649613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:52.649613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.649613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:52.649613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:52.649613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:52.650089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.650089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:52.650089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:52.650089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:52.650089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:52.650089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:52.650089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:52.650089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:52.650089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:52.650089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:52.650089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:52.650089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:52.650089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:52.650089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:52.650089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.650089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:52.650382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.650382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:52.650382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:52.650382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:52.650382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.650382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:52.650382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:52.650382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:45:52.650382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:52.650382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:52.650382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.650382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:52.650758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:52.650758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:52.650758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.650758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:52.650758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:52.650758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:52.650758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:52.650758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:52.650758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:52.650758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:52.650758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:52.650758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:52.650758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:52.650758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:52.651092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:52.651092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:52.651092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.651092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:52.651092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.651092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:52.651092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:52.651092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:52.651092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:52.651092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:52.651092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.651092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:52.651092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:52.651092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.651549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:52.651549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:52.651549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.651549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:52.651549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:52.651549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.651549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:52.651549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:52.651549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:45:52.651549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:52.651549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:52.651549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.651549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:52.651549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:52.651549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:52.651549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.651549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:52.651549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:52.651549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:52.652105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:52.652105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:52.652105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:52.652105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:52.652105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:52.652105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:52.652105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:52.652105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:52.652105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:52.652105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:52.652105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.652105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:52.652105 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:52.652105 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:52.652500 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:52.652500 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:52.652500 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:52.652500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.652500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:52.652500 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:52.652500 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:52.652500 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:52.652500 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:52.652500 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:52.652500 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.652500 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:52.652914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.652914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:52.652914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:52.652914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:52.652914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.652914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:52.652914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:52.652914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:52.652914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.652914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:52.652914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:52.652914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:52.652914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:52.652914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:52.652914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:52.653313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:52.653313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:52.653313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:52.653313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:52.653313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:52.653313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:52.653313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:52.653313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.653313 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:52.653781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.653781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:52.653781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:52.653781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:52.653781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.653781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:52.653781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:52.653781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:52.653781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:52.653781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.653781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:52.654114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:52.654114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:52.654114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.654114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:52.654114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:52.654114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:52.654114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:52.654114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:52.654114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:52.654114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:52.654114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:52.654114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:52.654114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:52.654114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:52.654521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:52.654521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:52.654521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.654521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:52.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:52.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:52.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:52.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:52.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:52.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:52.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:52.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:52.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:52.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:52.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:52.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:52.655277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:52.655277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:52.655277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:52.655277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.655277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:52.655277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:52.655277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:52.655277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.655277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:52.655277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:52.655277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:52.655277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:52.655277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:52.655277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:52.655277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:52.655277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:52.655277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:52.655277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:52.655277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:52.655646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:52.655646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:52.655646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.655646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:52.660360 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:52.660715 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:52.660832 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:52.660993 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:52.661317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:52.661556 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:52.661753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:52.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:52.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:52.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:52.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:52.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:52.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.662190 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:52.662889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.662889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:52.662889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:52.662889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:52.662889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.662889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:52.662889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:52.662889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.662889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:52.662889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:52.662889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:52.662889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.663095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:52.663095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:52.663095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:52.663095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:52.663095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:52.663095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:52.663095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:52.663095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:52.663095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:52.663095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:52.663095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:52.663095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:52.663095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:52.663349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.663349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:52.663684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.663684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:52.663684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:52.663684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:52.663684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:52.663684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:52.663684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.663684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:52.663684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:52.663684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.663684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:52.663684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:52.663684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.663684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:52.663684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:52.663684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:52.663957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:52.663957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:52.663957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.663957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:52.663957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:52.663957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:52.663957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:52.663957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:52.663957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:52.663957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:52.663957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:52.663957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:52.663957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:52.663957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:52.663957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:52.663957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:52.663957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:52.663957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:52.663957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:52.664310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:52.664310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:52.664310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:53.841560 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:53.842094 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:53.842094 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:53.842722 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:53.851611 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=240 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:45:53.875388 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:53.875622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:53.876159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:53.876159 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:53.876159 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:53.876159 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:53.876159 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:53.876159 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:53.876159 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:53.876159 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:53.876933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:53.876933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:53.876933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:53.876933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:53.876933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:53.876933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:53.876933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:53.876933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:53.876933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:53.876933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:53.876933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:53.876933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:53.877172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:53.877172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:53.877172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:53.877172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:53.877172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:53.877172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:53.877172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:53.877172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:53.877172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:53.877172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:53.877172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:53.877172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:53.877172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:53.877399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:53.877399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:53.877747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:53.877747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:53.877747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:53.877747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:53.877747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:53.877747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:53.877747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:53.877747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:53.877747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:53.877747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:53.877747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:53.877747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:53.877747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:53.877747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:53.877747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:53.877747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:53.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:53.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:53.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:53.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:53.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:53.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:53.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:53.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:53.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:53.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:53.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:53.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:53.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:53.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:53.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:53.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:53.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:53.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:53.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:53.878609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:53.878609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:53.878609 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:57.640600 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:57.641226 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:57.641226 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:57.641717 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:57.641899 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:57.642020 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:57.642146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.642666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:57.642666 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:57.642666 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:57.642666 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:57.642666 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:57.642666 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:57.642666 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.642666 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:57.643304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.643304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:57.643304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:57.643304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:57.643304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.643304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:57.643304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:57.643304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.643304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:57.643304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:57.643304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:57.643304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.643569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:57.643569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:57.643569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:57.643569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:57.643569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:57.643569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:57.643569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:57.643569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:57.643569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:57.643569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:57.643569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:57.643569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:57.643569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:57.643835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.643835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:57.644006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.644006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:57.644006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:57.644006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:57.644006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:57.644006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:57.644006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.644006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:57.644006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:57.644006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.644006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:57.644006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:57.644006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.644006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:57.644006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:57.644006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.644371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:45:57.644371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:57.644371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.644371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:57.644371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:57.644371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:57.644371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.644371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:57.644371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:57.644371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:57.644371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:57.644371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:57.644371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:57.644371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:57.644371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:57.644371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:57.644371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:57.644371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:57.644371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:57.644728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:57.644728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.644728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:57.646049 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:57.646306 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:57.646394 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:57.646479 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:45:57.646839 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:57.647040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.647478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:57.647478 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:57.647478 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:57.647478 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:57.647478 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:57.647478 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:57.647478 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.647478 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:57.648024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.648024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:57.648024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:57.648024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:57.648024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.648024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:57.648024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:57.648024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:57.648024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.648024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:57.648024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:57.648024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:57.648024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:57.648024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:57.648024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:57.648462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:57.648462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:57.648462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:57.648462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:57.648462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:57.648462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:57.648462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:57.648462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.648462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:57.648689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.648689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:57.648689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:57.648689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:57.648689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.648689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:57.648689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:57.648689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:45:57.648689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:57.648689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:57.648689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.648689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:57.648949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:57.648949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:57.648949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.648949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:57.648949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:57.648949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:57.648949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:57.648949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:57.648949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:57.648949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:57.648949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:57.648949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:57.648949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:57.648949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:57.649212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:57.649212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:57.649212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.649212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:57.649325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.649325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:57.649325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:57.649325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:57.649325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:57.649325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:57.649325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.649325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:57.649325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:57.649325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.649325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:57.649325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:57.649325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.649325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:57.649325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:57.649325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.649710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:45:57.649710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:45:57.649710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:45:57.649710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:57.649710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:57.649710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.649710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:57.649710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:57.649710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:57.649710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.649710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:57.649710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:57.649710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:57.649710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:57.649710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:57.649710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:57.649710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:57.649710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:57.650049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:57.650049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:57.650049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:57.650049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:57.650049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:57.650049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.650049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:57.650637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:57.650854 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:57.650974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:57.651112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:57.651430 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:57.651627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.652002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:57.652002 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:57.652002 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:57.652002 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:57.652002 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:57.652002 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:57.652002 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.652002 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:57.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:45:57.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:57.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:57.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:57.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:57.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:45:57.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:45:57.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:57.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:45:57.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:57.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:45:57.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:45:57.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:57.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:45:57.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:57.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:45:57.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:45:57.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:57.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:45:57.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:45:57.653594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.653594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:57.653594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:57.653594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:57.653594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.653594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:57.653594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:57.653594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:57.653594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:57.653594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.653594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:57.653953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:57.653953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:57.653953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.653953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:57.653953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:57.653953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:57.653953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:57.653953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:57.653953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:57.653953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:57.653953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:57.653953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:57.653953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:57.653953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:57.654479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:57.654479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:57.654479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.654479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:57.654740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.654740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:57.654740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:57.654740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:57.654740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:57.654740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:57.654740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.654740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:57.654740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:57.654740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.654740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:57.654740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:57.654740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.654740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:57.654740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:57.654740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.654740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:45:57.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:45:57.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:45:57.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:45:57.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:57.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:57.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:57.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:57.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:57.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:57.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:57.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:57.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:57.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:57.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:57.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:57.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:57.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:57.655392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:57.655392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:57.655392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.655392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:57.660758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:57.661043 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:57.661124 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:57.661270 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:57.661583 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:57.661771 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:57.661909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.662345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:57.662345 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:57.662345 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:57.662345 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:57.662345 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:57.662345 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:57.662345 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.662345 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:57.662950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.662950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:57.662950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:57.662950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:57.662950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.662950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:57.662950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:57.662950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.662950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:57.662950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:57.662950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:57.662950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.663319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:57.663319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:57.663319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:57.663319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:57.663319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:57.663319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:57.663319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:57.663319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:57.663319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:57.663319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:57.663319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:57.663319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:57.663319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:57.663667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.663667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:57.663667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.663667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:57.663667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:57.663667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:57.663667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:57.663667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:57.663667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.663667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:57.663667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:57.663667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.663667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:57.663667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:57.663667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.664076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:57.664076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:57.664076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:57.664076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:45:57.664076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:57.664076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.664076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:57.664076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:57.664076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:57.664076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:57.664076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:57.664076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:57.664076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:57.664076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:57.664076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:57.664076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:57.664076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:57.664076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:57.664076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:57.664550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:57.664550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:57.664550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:57.664550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:57.664550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:57.664550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:58.842602 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:58.843185 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:58.843185 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:58.844076 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:45:58.860930 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:45:58.877119 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:58.877364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:58.877884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:58.877884 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:58.877884 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:58.877884 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:58.877884 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:58.877884 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:58.877884 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:58.877884 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:58.878638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:58.878638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:58.878638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:58.878638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:45:58.878638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:58.878638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:58.878638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:58.878638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:58.878638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:58.878638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:58.878638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:58.878638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:58.878876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:58.878876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:58.878876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:58.878876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:58.878876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:58.878876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:58.878876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:58.878876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:58.878876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:58.878876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:58.878876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:58.878876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:58.878876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:58.879081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:58.879081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:58.879605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:58.879605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:45:58.879605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:45:58.879605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:45:58.879605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:45:58.879605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:45:58.879605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:58.879605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:45:58.879605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:45:58.879605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:58.879605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:45:58.879605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:45:58.879605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:45:58.879605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:45:58.879605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:45:58.879605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:45:58.880145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:45:58.880145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:58.880145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:58.880145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:58.880145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:58.880145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:45:58.880145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:58.880145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:45:58.880145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:58.880145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:45:58.880145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:58.880145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:45:58.880145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:45:58.880145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:58.880145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:45:58.880145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:58.880145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:45:58.880145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:45:58.880145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:58.880698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:45:58.880698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:58.880698 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:45:58.880698 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:45:58.880698 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:45:58.886086 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 08 14:45:58.886239 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Number of inconsistencies found at create/update: networks=5, routers=2, router_ports=2 {{(pid=89274) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 14:45:58.886403 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=5 {{(pid=89274) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 14:45:58.886498 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource a137566a-eaf1-4f5d-a192-15d63e6c049a (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:45:58.917342 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-a137566a-eaf1-4f5d-a192-15d63e6c049a, resource={'id': 'a137566a-eaf1-4f5d-a192-15d63e6c049a', 'name': 'private', 'tenant_id': 'ba473288446843d3a29e25267968ba61', 'project_id': 'ba473288446843d3a29e25267968ba61', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['7b3cb50f-b942-4054-b735-5f7be989b8f8', 'ba10f097-6c2b-4188-8d42-e9415eff0039'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:20:42Z', 'updated_at': '2026-06-08T14:20:44Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 26551}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:58.917670 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-a137566a-eaf1-4f5d-a192-15d63e6c049a, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:58.917971 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-c4b70b51-9d46-4e6b-a1e7-b3e6a6b8b666, external_ids_update=None, columns={'tag': 26551}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:58.918099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:58.918627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:58.918627 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:58.918627 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:58.918627 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:58.918627 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:58.918627 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:58.918627 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:58.918627 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:58.919463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource a137566a-eaf1-4f5d-a192-15d63e6c049a (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:58.919463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:58.919463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:45:58.919463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:45:58.919463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:45:58.919463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:45:58.919463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:45:58.919463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:58.919463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:58.919463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:58.919463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:58.919796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:58.919796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:58.919796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:58.919796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:58.919796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:58.919796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:58.919796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:58.919796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:58.919796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:58.919796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:58.919796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:58.919796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:58.919796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:58.919796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:58.920020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:58.920020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:58.920020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:58.920020 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource fffd4e9d-b343-4a97-8e5f-4015f805fc46 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:45:58.952794 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, resource={'id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'name': 'public', 'tenant_id': 'ba473288446843d3a29e25267968ba61', 'project_id': 'ba473288446843d3a29e25267968ba61', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['4ee83802-2181-432f-a76d-d3335bbf4e0b', 'b9859e49-8026-492d-a0a4-11326477713e'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-08T14:20:46Z', 'updated_at': '2026-06-08T14:20:52Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:58.953023 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:58.953533 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-0817ce61-1c85-4457-911a-0b8fdccf389d, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:58.953533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:58.954073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:58.954073 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:58.954073 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:58.954073 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:58.954073 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:58.954073 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:58.954073 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:58.954073 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:58.954925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource fffd4e9d-b343-4a97-8e5f-4015f805fc46 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:58.954925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:58.954925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:45:58.954925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:45:58.954925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:45:58.954925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:45:58.954925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:45:58.954925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:58.954925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:58.954925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:58.954925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:58.955323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:58.955323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:58.955323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:58.955323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:58.955323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:58.955323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:58.955323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:58.955323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:58.955323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:58.955323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:58.955323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:58.955323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:58.955323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:58.955323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:58.955768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:58.955768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:58.955768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:58.955768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 73426422-a7fc-44cd-9fc5-cd2d8fa95c38 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:45:58.987793 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-73426422-a7fc-44cd-9fc5-cd2d8fa95c38, resource={'id': '73426422-a7fc-44cd-9fc5-cd2d8fa95c38', 'name': 'shared', 'tenant_id': 'ba473288446843d3a29e25267968ba61', 'project_id': 'ba473288446843d3a29e25267968ba61', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['2b53474a-74e5-4c44-bb1b-86c8215bd7ab'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:21:52Z', 'updated_at': '2026-06-08T14:21:52Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 292}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:58.988072 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-73426422-a7fc-44cd-9fc5-cd2d8fa95c38, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:58.988383 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-5c6a587e-b2f4-42c7-9706-8a2e20b0df46, external_ids_update=None, columns={'tag': 292}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:58.988462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:58.989018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:58.989018 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:58.989018 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:58.989018 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:58.989018 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:58.989018 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:58.989018 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:58.989018 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:58.990107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 73426422-a7fc-44cd-9fc5-cd2d8fa95c38 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:58.990107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:58.990107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:45:58.990107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:45:58.990107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:45:58.990107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:45:58.990107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:45:58.990107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:58.990107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:58.990107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:58.990107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:58.990546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:58.990546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:58.990546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:58.990546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:58.990546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:58.990546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:58.990546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:58.990546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:58.990546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:58.990546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:58.990546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:58.990546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:58.990546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:58.990546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:58.990950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:58.990950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:58.990950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:58.990950 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 24b6ed09-1bb3-4254-a10d-828b7d27f5ff (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:45:59.016352 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-24b6ed09-1bb3-4254-a10d-828b7d27f5ff, resource={'id': '24b6ed09-1bb3-4254-a10d-828b7d27f5ff', 'name': 'tempest-TestVolumeBootPattern-626079824-network', 'tenant_id': '2eaebe9864e04d018c4fc6f6678a999e', 'project_id': '2eaebe9864e04d018c4fc6f6678a999e', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['705c8961-baad-406e-ab5c-2201ea407fe2'], 'standard_attr_id': 1681, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:42:04Z', 'updated_at': '2026-06-08T14:42:04Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 2940}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.016649 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-24b6ed09-1bb3-4254-a10d-828b7d27f5ff, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestVolumeBootPattern-626079824-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.016949 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-6e5ddab1-8110-4b81-9897-eecf8025269d, external_ids_update=None, columns={'tag': 2940}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.017062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.017752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:59.017752 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:59.017752 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:59.017752 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:59.017752 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:59.017752 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:59.017752 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.017752 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:59.018788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 24b6ed09-1bb3-4254-a10d-828b7d27f5ff (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.018788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:59.018788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:45:59.018788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:45:59.018788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:45:59.018788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:45:59.018788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:45:59.018788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:59.018788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:59.018788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:59.018788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:59.019036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:59.019036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:59.019036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:59.019036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:59.019036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:59.019036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:59.019036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:59.019036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:59.019036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:59.019036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:59.019036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:59.019036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:59.019036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:59.019036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:59.019286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:59.019286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.019286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:59.019286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 3eeaf3a4-bc2e-45fa-a8dd-1ed62594f914 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:45:59.045674 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-3eeaf3a4-bc2e-45fa-a8dd-1ed62594f914, resource={'id': '3eeaf3a4-bc2e-45fa-a8dd-1ed62594f914', 'name': 'tempest-TestVolumeBootPattern-756877900-network', 'tenant_id': '1156286d14d248bc81f670d575137b7d', 'project_id': '1156286d14d248bc81f670d575137b7d', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['99e3c1b8-f5e7-4f4e-973a-1bf722b09cb1'], 'standard_attr_id': 1693, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:42:08Z', 'updated_at': '2026-06-08T14:42:08Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 1580}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.045946 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-3eeaf3a4-bc2e-45fa-a8dd-1ed62594f914, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestVolumeBootPattern-756877900-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.046241 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-4bee6417-69a0-4600-946b-d17332049503, external_ids_update=None, columns={'tag': 1580}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.046374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.047054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:59.047054 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:59.047054 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:59.047054 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:59.047054 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:59.047054 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:59.047054 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.047054 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:59.048033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 3eeaf3a4-bc2e-45fa-a8dd-1ed62594f914 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.048033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:59.048033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:45:59.048033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:45:59.048033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:45:59.048033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:45:59.048033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:45:59.048033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:59.048033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:59.048033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:59.048033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:59.048425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:59.048425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:59.048425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:59.048425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:59.048425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:59.048425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:59.048425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:59.048425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:59.048425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:59.048425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:59.048425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:59.048425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:59.048425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:59.048425 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:59.048815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:59.048815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.048815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:59.048815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 1e6d0d4f-769b-40ee-9f46-dc2dd219205c (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:45:59.125657 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, resource={'id': '1e6d0d4f-769b-40ee-9f46-dc2dd219205c', 'name': 'tempest-TestVolumeBootPattern-626079824-router', 'project_id': '2eaebe9864e04d018c4fc6f6678a999e', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.12'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::309'}], 'enable_snat': True}, 'gw_port_id': '684225f0-1038-4501-9c7d-cb064d811b41', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.12', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::309', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:42:05Z', 'updated_at': '2026-06-08T14:42:07Z', 'revision_number': 3, 'tenant_id': '2eaebe9864e04d018c4fc6f6678a999e'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.125911 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, columns={'external_ids': {'neutron:router_name': 'tempest-TestVolumeBootPattern-626079824-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.126087 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "684225f0-1038-4501-9c7d-cb064d811b41"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.126227 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "684225f0-1038-4501-9c7d-cb064d811b41"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.126358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.127099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:59.127099 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:59.127099 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:59.127099 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:59.127099 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:59.127099 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:59.127099 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.127099 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:59.127356 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router 1e6d0d4f-769b-40ee-9f46-dc2dd219205c. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.128843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 1e6d0d4f-769b-40ee-9f46-dc2dd219205c (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.128843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:59.128843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:45:59.128843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:45:59.128843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:45:59.128843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:45:59.128843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:45:59.128843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:45:59.128843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:45:59.128843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:45:59.128843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:45:59.129154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:45:59.129154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:45:59.129154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:59.129154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:59.129154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:59.129154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:59.129154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:59.129154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:59.129154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:59.129154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:59.129154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:59.129154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:59.129154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:59.129154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:59.129417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:59.129417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:59.129417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:59.129417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:59.129417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:59.129417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:59.129417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:59.129417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.129417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:59.129417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 77ab8ddd-dee1-41ae-b38b-83c0f140f13d (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:45:59.209277 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, resource={'id': '77ab8ddd-dee1-41ae-b38b-83c0f140f13d', 'name': 'tempest-TestVolumeBootPattern-756877900-router', 'project_id': '1156286d14d248bc81f670d575137b7d', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.188'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::2eb'}], 'enable_snat': True}, 'gw_port_id': 'affc847c-00fe-402e-9933-672b6b2f715d', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.188', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::2eb', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:42:09Z', 'updated_at': '2026-06-08T14:42:10Z', 'revision_number': 3, 'tenant_id': '1156286d14d248bc81f670d575137b7d'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.209530 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, columns={'external_ids': {'neutron:router_name': 'tempest-TestVolumeBootPattern-756877900-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.209798 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "affc847c-00fe-402e-9933-672b6b2f715d"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.209952 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "affc847c-00fe-402e-9933-672b6b2f715d"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.210086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.210783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:59.210783 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:59.210783 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:59.210783 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:59.210783 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:59.210783 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:59.210783 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.210783 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:59.211025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router 77ab8ddd-dee1-41ae-b38b-83c0f140f13d. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.212674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 77ab8ddd-dee1-41ae-b38b-83c0f140f13d (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.212674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:59.212674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:45:59.212674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:45:59.212674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:45:59.212674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:45:59.212674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:45:59.212674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:45:59.212674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:45:59.212674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:45:59.212674 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:45:59.212970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:45:59.212970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:45:59.212970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:59.212970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:59.212970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:59.212970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:59.212970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:59.212970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:59.212970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:59.212970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:59.212970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:59.212970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:59.212970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:59.212970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:59.213273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:59.213273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:59.213273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:59.213273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:59.213273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:59.213273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:59.213273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:59.213273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.213273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:59.213273 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 684225f0-1038-4501-9c7d-cb064d811b41 (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:45:59.310825 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-684225f0-1038-4501-9c7d-cb064d811b41, resource={'id': '684225f0-1038-4501-9c7d-cb064d811b41', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:93:99:3d', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '1e6d0d4f-769b-40ee-9f46-dc2dd219205c', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1686, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.12'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::309'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:42:05Z', 'updated_at': '2026-06-08T14:42:07Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.311225 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-684225f0-1038-4501-9c7d-cb064d811b41, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::309/64', '172.24.5.12/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.311540 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=684225f0-1038-4501-9c7d-cb064d811b41, lrouter_port=lrp-684225f0-1038-4501-9c7d-cb064d811b41, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.311804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.312484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:59.312484 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:59.312484 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:59.312484 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:59.312484 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:59.312484 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:59.312484 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.312484 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:59.313327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 684225f0-1038-4501-9c7d-cb064d811b41 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.313327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:59.313327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:45:59.313327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:45:59.313327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:45:59.313327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:45:59.313327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:45:59.313327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:59.313327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:59.313327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:59.313327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:59.313622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:59.313622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:59.313622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:59.313622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:59.313622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:59.313622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:59.313622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:59.313622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:59.313622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:59.313622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:59.313622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:59.313622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:59.313622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:59.313622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:59.313832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:59.313832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.313832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:59.313832 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource affc847c-00fe-402e-9933-672b6b2f715d (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:45:59.414513 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-affc847c-00fe-402e-9933-672b6b2f715d, resource={'id': 'affc847c-00fe-402e-9933-672b6b2f715d', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:4d:b8:2c', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '77ab8ddd-dee1-41ae-b38b-83c0f140f13d', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1698, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.188'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::2eb'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:42:09Z', 'updated_at': '2026-06-08T14:42:13Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.414830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-affc847c-00fe-402e-9933-672b6b2f715d, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::2eb/64', '172.24.5.188/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.415117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=affc847c-00fe-402e-9933-672b6b2f715d, lrouter_port=lrp-affc847c-00fe-402e-9933-672b6b2f715d, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:45:59.415354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.415932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:45:59.415932 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:59.415932 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:45:59.415932 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:45:59.415932 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:59.415932 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:45:59.415932 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.415932 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:45:59.416892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource affc847c-00fe-402e-9933-672b6b2f715d (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.416892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:45:59.416892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:45:59.416892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:45:59.416892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:45:59.416892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:45:59.416892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:45:59.416892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:45:59.416892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:59.416892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:59.416892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:45:59.417219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:45:59.417219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:45:59.417219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:45:59.417219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:45:59.417219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:45:59.417219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:45:59.417219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:45:59.417219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:45:59.417219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:45:59.417219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:45:59.417219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:45:59.417219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:45:59.417219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:45:59.417219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:45:59.417637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:45:59.417637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:45:59.417637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:45:59.417637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 2d70280f-c1ff-40ec-95b8-6a1bcc85f1c3 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:45:59.417637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(2d70280f-c1ff-40ec-95b8-6a1bcc85f1c3) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:45:59.422128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 34de5e3d-8039-4618-91f1-9d1c0f30817b (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:45:59.422351 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(34de5e3d-8039-4618-91f1-9d1c0f30817b) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:45:59.427646 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 954cde3a-43db-4e15-8237-a1c06f71bb98 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:45:59.427890 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(954cde3a-43db-4e15-8237-a1c06f71bb98) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:45:59.432769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource d0d73269-e7e3-4594-b8c4-35c9e324d743 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:45:59.433041 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(d0d73269-e7e3-4594-b8c4-35c9e324d743) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:45:59.437543 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource f074009d-7c60-42b1-af8e-fa1880038db9 (type: router_ports) at delete {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 14:45:59.437833 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] delete_revision(f074009d-7c60-42b1-af8e-fa1880038db9) {{(pid=89274) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 14:45:59.442499 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_for_inconsistencies finished in 0.562 seconds Jun 08 14:46:02.641504 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:02.641948 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:02.642003 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:02.642732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:02.642885 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:02.642998 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:02.643101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.643640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:02.643640 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:02.643640 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:02.643640 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:02.643640 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:02.643640 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:02.643640 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.643640 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:02.644460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.644460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:02.644460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:02.644460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:02.644460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.644460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:02.644460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:02.644460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.644460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:02.644460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:02.644460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:02.644460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.644853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:02.644853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:02.644853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:02.644853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:02.644853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:02.644853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:02.644853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:02.644853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:02.644853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:02.644853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:02.644853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:02.644853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:02.644853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:02.645237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.645237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:02.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:02.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:02.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:02.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:02.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:02.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:02.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:02.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:02.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:02.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:02.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:02.645554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.645883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:02.645883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:02.645883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.645883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:02.645883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:02.645883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:02.645883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.645883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:02.645883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:02.645883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:02.645883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:02.645883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:02.645883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:02.645883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:02.645883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:02.645883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:02.645883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:02.645883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:02.645883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:02.646216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:02.646216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.646216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:02.646661 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:02.646871 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:02.646992 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:02.647127 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:46:02.647427 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:02.647592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.647952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:02.647952 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:02.647952 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:02.647952 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:02.647952 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:02.647952 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:02.647952 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.647952 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:02.648619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.648619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:02.648619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:02.648619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:02.648619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.648619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:02.648619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:02.648619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:02.648619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.648619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:02.648619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:02.648619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:02.648619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:02.648619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:02.648619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:02.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:02.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:02.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:02.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:02.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:02.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:02.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:02.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.649044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:02.649520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.649520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:02.649520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:02.649520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:02.649520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.649520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:02.649520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:02.649520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:46:02.649520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:02.649520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:02.649520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.649520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:02.649874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:02.649874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:02.649874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.649874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:02.649874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:02.649874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:02.649874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:02.649874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:02.649874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:02.649874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:02.649874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:02.649874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:02.649874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:02.649874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:02.650301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:02.650301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:02.650301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.650301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:02.650676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.650676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:02.650676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:02.650676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:02.650676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:02.650676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:02.650676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.650676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:02.650676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:02.650676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.650676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:02.650676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:02.650676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.650676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:02.650676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:02.650676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.651102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:02.651102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:02.651102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:46:02.651102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:02.651102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:02.651102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.651102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:02.651102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:02.651102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:02.651102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.651102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:02.651102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:02.651102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:02.651102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:02.651102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:02.651102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:02.651102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:02.651102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:02.651575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:02.651575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:02.651575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:02.651575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:02.651575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:02.651575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.651575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:02.651575 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:02.651575 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:02.651575 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:02.651769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:02.651769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:02.651843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.652225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:02.652225 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:02.652225 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:02.652225 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:02.652225 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:02.652225 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:02.652225 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.652225 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:02.652877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.652877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:02.652877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:02.652877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:02.652877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.652877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:02.652877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:02.652877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:02.652877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.652877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:02.652877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:02.652877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:02.652877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:02.652877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:02.652877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:02.653405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:02.653405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:02.653405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:02.653405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:02.653405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:02.653405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:02.653405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:02.653405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.653405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:02.653901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.653901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:02.653901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:02.653901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:02.653901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.653901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:02.653901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:02.653901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:02.653901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:02.653901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.653901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:02.654234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:02.654234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:02.654234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.654234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:02.654234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:02.654234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:02.654234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:02.654234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:02.654234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:02.654234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:02.654234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:02.654234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:02.654234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:02.654234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:02.655684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:02.655684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:02.655684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.655684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:02.655684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.655684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:02.655684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:02.655684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:02.655684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:02.655684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:02.655684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.655684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:02.655684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:02.655684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.656206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:02.656206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:02.656206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.656206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:02.656206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:02.656206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.656206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:02.656206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:02.656206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:02.656206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:02.656206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.656206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:02.656206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:02.656206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:02.656206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.656206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:02.656206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:02.656206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:02.656206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:02.656818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:02.656818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:02.656818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:02.656818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:02.656818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:02.656818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:02.656818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:02.656818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:02.656818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:02.656818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.656818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:02.663748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:02.663961 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:02.664073 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:02.664262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:02.664541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:02.664770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:02.664927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.665365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:02.665365 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:02.665365 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:02.665365 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:02.665365 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:02.665365 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:02.665365 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.665365 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:02.666269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.666269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:02.666269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:02.666269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:02.666269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.666269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:02.666269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:02.666269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.666269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:02.666269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:02.666269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:02.666269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.666542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:02.666542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:02.666542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:02.666542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:02.666542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:02.666542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:02.666542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:02.666542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:02.666542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:02.666542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:02.666542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:02.666542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:02.666542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:02.666777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.666777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:02.666777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.666777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:02.666777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:02.666777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:02.666777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:02.666777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:02.666777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.666777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:02.666777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:02.666777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.666777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:02.666777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:02.666777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.667661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:02.667661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:02.667661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:02.667661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:02.667661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:02.667661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.667661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:02.667661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:02.667661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:02.667661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:02.667661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:02.667661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:02.667661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:02.667661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:02.667661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:02.667661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:02.667661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:02.667661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:02.667661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:02.667978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:02.667978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:02.667978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:02.667978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:02.667978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:02.667978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:03.843565 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:03.844069 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:03.844116 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:03.844765 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:03.879594 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:03.879789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:03.880213 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:03.880213 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:03.880213 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:03.880213 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:03.880213 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:03.880213 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:03.880213 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:03.880213 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:03.880809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:03.880809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:03.880809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:03.880809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:03.880809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:03.880809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:03.880809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:03.880809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:03.880809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:03.880809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:03.880809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:03.880809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:03.881075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:03.881075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:03.881075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:03.881075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:03.881075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:03.881075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:03.881075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:03.881075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:03.881075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:03.881075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:03.881075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:03.881075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:03.881075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:03.881330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:03.881330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:03.881582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:03.881582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:03.881582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:03.881582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:03.881582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:03.881582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:03.881582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:03.881582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:03.881582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:03.881582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:03.881582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:03.881582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:03.881582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:03.881582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:03.881582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:03.881582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:03.881871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:03.881871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:03.881871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:03.881871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:03.881871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:03.881871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:03.881871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:03.881871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:03.881871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:03.881871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:03.881871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:03.881871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:03.881871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:03.881871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:03.881871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:03.881871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:03.881871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:03.881871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:03.881871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:03.882175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:03.882175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:03.882175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:07.642042 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:07.642747 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:07.642747 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:07.643203 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:07.643408 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:07.643542 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:07.643725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.644185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:07.644185 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:07.644185 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:07.644185 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:07.644185 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:07.644185 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:07.644185 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.644185 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:07.644893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.644893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:07.644893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:07.644893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:07.644893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.644893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:07.644893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:07.644893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.644893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:07.644893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:07.644893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:07.644893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.645397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:07.645397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:07.645397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:07.645397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:07.645397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:07.645397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:07.645397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:07.645397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:07.645397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:07.645397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:07.645397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:07.645397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:07.645397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:07.645868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.645868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:07.645868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.645868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:07.645868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:07.645868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:07.645868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:07.645868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:07.645868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.645868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:07.645868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:07.645868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.645868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:07.645868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:07.645868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.646376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:07.646376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:07.646376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.646376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:07.646376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:07.646376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.646376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:07.646376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:07.646376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:07.646376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.646376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:07.646376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:07.646376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:07.646376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:07.646376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:07.646376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:07.646376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:07.646376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:07.646376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:07.646816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:07.646816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:07.646816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:07.646816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:07.646816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.646816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:07.647056 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:07.647398 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:07.647548 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:07.647707 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:46:07.648095 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:07.648325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.648797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:07.648797 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:07.648797 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:07.648797 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:07.648797 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:07.648797 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:07.648797 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.648797 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:07.649517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.649517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:07.649517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:07.649517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:07.649517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.649517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:07.649517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:07.649517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:07.649517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.649517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:07.649517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:07.649517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:07.649517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:07.649517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:07.649517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:07.650073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:07.650073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:07.650073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:07.650073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:07.650073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:07.650073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:07.650073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:07.650073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.650073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:07.650558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.650558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:07.650558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:07.650558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:07.650558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.650558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:07.650558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:07.650558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:46:07.650558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:07.650558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:07.650558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.650558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:07.650980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:07.650980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:07.650980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.650980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:07.650980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:07.650980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:07.650980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:07.650980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:07.650980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:07.650980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:07.650980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:07.650980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:07.650980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:07.650980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:07.651335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:07.651335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:07.651335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.651335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:07.651868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.651868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:07.651868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:07.651868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:07.651868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:07.651868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:07.651868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.651868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:07.651868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:07.651868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.651868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:07.651868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:07.651868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.651868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:07.651868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:07.651868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.652423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:07.652423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:07.652423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:46:07.652423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:07.652423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:07.652423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.652423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:07.652423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:07.652423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:07.652423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.652423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:07.652423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:07.652423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:07.652423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:07.652423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:07.652423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:07.652423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:07.652423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:07.653044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:07.653044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:07.653044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:07.653044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:07.653044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:07.653044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.653044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:07.653044 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:07.653044 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:07.653044 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:07.653408 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:07.653408 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:07.653408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.653873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:07.653873 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:07.653873 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:07.653873 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:07.653873 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:07.653873 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:07.653873 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.653873 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:07.654646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.654646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:07.654646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:07.654646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:07.654646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.654646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:07.654646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:07.654646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:07.654646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.654646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:07.654646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:07.654646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:07.654646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:07.654646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:07.654646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:07.655104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:07.655104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:07.655104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:07.655104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:07.655104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:07.655104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:07.655104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:07.655104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.655104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:07.655638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.655638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:07.655638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:07.655638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:07.655638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.655638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:07.655638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:07.655638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:07.655638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:07.655638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.655638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:07.656043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:07.656043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:07.656043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.656043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:07.656043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:07.656043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:07.656043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:07.656043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:07.656043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:07.656043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:07.656043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:07.656043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:07.656043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:07.656043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:07.656462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:07.656462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:07.656462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.656462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:07.656767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.656767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:07.656767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:07.656767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:07.656767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:07.656767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:07.656767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.656767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:07.656767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:07.656767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.656767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:07.656767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:07.656767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.656767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:07.656767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:07.656767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.656767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:07.657138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:07.657138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:07.657138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:07.657138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.657138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:07.657138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:07.657138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:07.657138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.657138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:07.657138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:07.657138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:07.657138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:07.657138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:07.657138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:07.657138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:07.657138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:07.657138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:07.657138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:07.657138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:07.657447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:07.657447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:07.657447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.657447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:07.664068 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:07.664354 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:07.664492 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:07.664704 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:07.665077 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:07.665334 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:07.665499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.665967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:07.665967 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:07.665967 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:07.665967 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:07.665967 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:07.665967 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:07.665967 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.665967 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:07.666744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.666744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:07.666744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:07.666744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:07.666744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.666744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:07.666744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:07.666744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.666744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:07.666744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:07.666744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:07.666744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.667146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:07.667146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:07.667146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:07.667146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:07.667146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:07.667146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:07.667146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:07.667146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:07.667146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:07.667146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:07.667146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:07.667146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:07.667146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:07.667560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.667560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:07.667854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.667854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:07.667854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:07.667854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:07.667854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:07.667854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:07.667854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.667854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:07.667854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:07.667854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.667854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:07.667854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:07.667854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.667854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:07.667854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:07.667854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:07.668355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:07.668355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:07.668355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.668355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:07.668355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:07.668355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:07.668355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:07.668355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:07.668355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:07.668355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:07.668355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:07.668355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:07.668355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:07.668355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:07.668355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:07.668355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:07.668355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:07.668355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:07.668355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:07.668912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:07.668912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:07.668912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:08.844175 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:08.844775 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:08.844775 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:08.845156 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:09.117279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:09.117456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:09.118096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:09.118096 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:09.118096 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:09.118096 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:09.118096 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:09.118096 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:09.118096 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:09.118096 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:09.119347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.274 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:09.119347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:09.119347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:09.119347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:09.119347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:09.119347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:09.119347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:09.119347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:09.119347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:09.119347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:09.119347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:09.119347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:09.119883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:09.119883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:09.119883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:09.119883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:09.119883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:09.119883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:09.119883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:09.119883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:09.119883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:09.119883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:09.119883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:09.119883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:09.119883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:09.120204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:09.120204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:09.120616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:09.120616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:09.120616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:09.120616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:09.120616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:09.120616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:09.120616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:09.120616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:09.120616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:09.120616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:09.120616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:09.120616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:09.120616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:09.120616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:09.120616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:09.120616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:09.120901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:09.120901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:09.120901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:09.120901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:09.120901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:09.120901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:09.120901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:09.120901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:09.120901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:09.120901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:09.120901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:09.120901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:09.120901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:09.120901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:09.120901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:09.120901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:09.120901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:09.120901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:09.120901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:09.121187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:09.121187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:09.121187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:12.642560 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:12.643111 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:12.643111 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:12.643550 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:12.643699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:12.643821 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:12.643909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.644378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:12.644378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:12.644378 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:12.644378 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:12.644378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:12.644378 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:12.644378 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.644378 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:12.645156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.645156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:12.645156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:12.645156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:12.645156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.645156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:12.645156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:12.645156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.645156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:12.645156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:12.645156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:12.645156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.645417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:12.645417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:12.645417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:12.645417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:12.645417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:12.645417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:12.645417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:12.645417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:12.645417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:12.645417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:12.645417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:12.645417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:12.645417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:12.645643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.645643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:12.646018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.646018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:12.646018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:12.646018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:12.646018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:12.646018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:12.646018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.646018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:12.646018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:12.646018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.646018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:12.646018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:12.646018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.646018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:12.646018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:12.646018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.646339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:12.646339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:12.646339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.646339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:12.646339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:12.646339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:12.646339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.646339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:12.646339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:12.646339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:12.646339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:12.646339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:12.646339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:12.646339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:12.646339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:12.646339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:12.646339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:12.646339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:12.646339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:12.646702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:12.646702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.646702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:12.647465 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:12.647733 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:12.647822 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:12.647909 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:46:12.648187 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:12.648379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.648759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:12.648759 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:12.648759 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:12.648759 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:12.648759 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:12.648759 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:12.648759 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.648759 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:12.649367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.649367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:12.649367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:12.649367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:12.649367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.649367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:12.649367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:12.649367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:12.649367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.649367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:12.649367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:12.649367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:12.649367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:12.649367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:12.649367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:12.649728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:12.649728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:12.649728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:12.649728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:12.649728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:12.649728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:12.649728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:12.649728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.649728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:12.650075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.650075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:12.650075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:12.650075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:12.650075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.650075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:12.650075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:12.650075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:46:12.650075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:12.650075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:12.650075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.650075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:12.650777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:12.650777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:12.650777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.650777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:12.650777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:12.650777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:12.650777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:12.650777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:12.650777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:12.650777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:12.650777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:12.650777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:12.650777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:12.650777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:12.651329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:12.651329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:12.651329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.651329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:12.651329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.651329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:12.651329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:12.651329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:12.651329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:12.651329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:12.651329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.651329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:12.651329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:12.651329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.651559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:12.651559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:12.651559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.651559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:12.651559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:12.651559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.651559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:12.651559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:12.651559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:46:12.651559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:12.651559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:12.651559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.651559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:12.651559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:12.651559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:12.651559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.651559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:12.651559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:12.651559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:12.651889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:12.651889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:12.651889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:12.651889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:12.651889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:12.651889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:12.651889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:12.651889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:12.651889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:12.651889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:12.651889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.651889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:12.652552 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:12.652815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:12.652925 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:12.653057 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:12.653364 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:12.653607 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.653965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:12.653965 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:12.653965 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:12.653965 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:12.653965 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:12.653965 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:12.653965 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.653965 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:12.654720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.654720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:12.654720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:12.654720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:12.654720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.654720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:12.654720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:12.654720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:12.654720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.654720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:12.654720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:12.654720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:12.654720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:12.654720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:12.654720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:12.655348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:12.655348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:12.655348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:12.655348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:12.655348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:12.655348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:12.655348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:12.655348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.655348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:12.655776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.655776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:12.655776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:12.655776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:12.655776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.655776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:12.655776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:12.655776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:12.655776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:12.655776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.655776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:12.656187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:12.656187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:12.656187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.656187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:12.656187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:12.656187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:12.656187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:12.656187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:12.656187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:12.656187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:12.656187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:12.656187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:12.656187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:12.656187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:12.656620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:12.656620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:12.656620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.656620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:12.656789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.656789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:12.656789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:12.656789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:12.656789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:12.656789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:12.656789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.656789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:12.656789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:12.656789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.656789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:12.656789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:12.656789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.656789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:12.656789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:12.656789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.656789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:12.657372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:12.657372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:12.657372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:12.657372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.657372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:12.657372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:12.657372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:12.657372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.657372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:12.657372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:12.657372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:12.657372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:12.657372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:12.657372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:12.657372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:12.657372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:12.657372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:12.657372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:12.657372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:12.657718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:12.657718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:12.657718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.657718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:12.664552 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:12.664887 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:12.665007 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:12.665135 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:12.665579 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:12.665917 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:12.666068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.666693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:12.666693 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:12.666693 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:12.666693 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:12.666693 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:12.666693 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:12.666693 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.666693 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:12.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:12.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:12.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:12.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:12.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:12.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:12.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:12.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:12.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.667737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:12.667737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:12.667737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:12.667737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:12.667737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:12.667737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:12.667737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:12.667737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:12.667737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:12.667737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:12.667737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:12.667737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:12.667737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:12.667974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.667974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:12.668156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.668156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:12.668156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:12.668156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:12.668156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:12.668156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:12.668156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.668156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:12.668156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:12.668156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.668156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:12.668156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:12.668156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.668156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:12.668156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:12.668156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:12.668835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:12.668835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:12.668835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.668835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:12.668835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:12.668835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:12.668835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:12.668835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:12.668835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:12.668835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:12.668835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:12.668835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:12.668835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:12.668835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:12.668835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:12.668835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:12.668835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:12.668835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:12.668835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:12.669492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:12.669492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:12.669492 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:13.844560 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:13.845153 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:13.845153 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:13.846105 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:13.878756 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:13.878756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:13.879134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:13.879134 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:13.879134 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:13.879134 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:13.879134 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:13.879134 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:13.879134 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:13.879134 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:13.879903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:13.879903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:13.879903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:13.879903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:13.879903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:13.879903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:13.879903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:13.879903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:13.879903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:13.879903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:13.879903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:13.879903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:13.880144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:13.880144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:13.880144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:13.880144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:13.880144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:13.880144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:13.880144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:13.880144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:13.880144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:13.880144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:13.880144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:13.880144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:13.880144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:13.880393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:13.880393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:13.880962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:13.880962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:13.880962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:13.880962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:13.880962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:13.880962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:13.880962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:13.880962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:13.880962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:13.880962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:13.880962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:13.880962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:13.880962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:13.880962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:13.880962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:13.880962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:13.881241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:13.881241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:13.881241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:13.881241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:13.881241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:13.881241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:13.881241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:13.881241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:13.881241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:13.881241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:13.881241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:13.881241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:13.881241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:13.881241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:13.881241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:13.881241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:13.881241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:13.881241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:13.881241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:13.881671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:13.881671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:13.881671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:17.643047 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:17.643530 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:17.643530 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:17.644179 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:17.644404 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:17.644537 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:17.644663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.645089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:17.645089 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:17.645089 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:17.645089 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:17.645089 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:17.645089 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:17.645089 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.645089 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:17.645761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.645761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:17.645761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:17.645761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:17.645761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.645761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:17.645761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:17.645761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.645761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:17.645761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:17.645761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:17.645761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.646056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:17.646056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:17.646056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:17.646056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:17.646056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:17.646056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:17.646056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:17.646056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:17.646056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:17.646056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:17.646056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:17.646056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:17.646056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:17.646517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.646517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:17.646517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.646517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:17.646517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:17.646517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:17.646517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:17.646517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:17.646517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.646517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:17.646517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:17.646517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.646517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:17.646517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:17.646517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.646865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:17.646865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:17.646865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.646865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:17.646865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:17.646865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.646865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:17.646865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:17.646865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:17.646865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.646865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:17.646865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:17.646865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:17.646865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:17.646865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:17.646865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:17.646865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:17.646865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:17.646865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:17.647267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:17.647267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:17.647267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:17.647267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:17.647267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.647267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:17.647708 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:17.647894 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:17.648013 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:17.648140 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:46:17.648468 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:17.648642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.649034 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:17.649670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.649670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:17.649670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:17.649670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:17.649670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.649670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:17.649670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:17.649670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:17.649670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.649670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:17.649670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:17.649670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:17.649670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:17.649670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:17.649670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:17.650189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:17.650189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:17.650189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:17.650189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:17.650189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:17.650189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:17.650189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:17.650189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.650189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:17.650545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.650545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:17.650545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:17.650545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:17.650545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.650545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:17.650545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:17.650545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:46:17.650545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:17.650545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:17.650545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.650545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:17.650922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:17.650922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:17.650922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.650922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:17.650922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:17.650922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:17.650922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:17.650922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:17.650922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:17.650922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:17.650922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:17.650922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:17.650922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:17.650922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:17.651370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:17.651370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:17.651370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.651370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:17.651617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.651617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:17.651617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:17.651617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:17.651617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:17.651617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:17.651617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.651617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:17.651617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:17.651617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.651617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:17.651617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:17.651617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.651617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:17.651617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:17.651617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.651955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:17.651955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:17.651955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:46:17.651955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:17.651955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:17.651955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.651955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:17.651955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:17.651955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:17.651955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.651955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:17.651955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:17.651955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:17.651955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:17.651955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:17.651955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:17.651955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:17.651955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:17.652273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:17.652273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:17.652273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:17.652273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:17.652273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:17.652273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.652273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:17.652807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:17.652991 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:17.653113 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:17.653244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:17.653755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:17.653804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.654188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:17.654188 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:17.654188 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:17.654188 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:17.654188 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:17.654188 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:17.654188 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.654188 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:17.654826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.654826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:17.654826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:17.654826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:17.654826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.654826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:17.654826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:17.654826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:17.654826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.654826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:17.654826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:17.654826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:17.654826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:17.654826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:17.654826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:17.655282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:17.655282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:17.655282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:17.655282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:17.655282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:17.655282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:17.655282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:17.655282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.655282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:17.655745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.655745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:17.655745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:17.655745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:17.655745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.655745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:17.655745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:17.655745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:17.655745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:17.655745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.655745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:17.656091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:17.656091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:17.656091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.656091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:17.656091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:17.656091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:17.656091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:17.656091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:17.656091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:17.656091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:17.656091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:17.656091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:17.656091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:17.656091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:17.656528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:17.656528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:17.656528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.656528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:17.656858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.656858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:17.656858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:17.656858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:17.656858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:17.656858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:17.656858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.656858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:17.656858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:17.656858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.656858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:17.656858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:17.656858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.656858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:17.656858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:17.656858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.656858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:17.657202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:17.657202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:17.657202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:17.657202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.657202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:17.657202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:17.657202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:17.657202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.657202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:17.657202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:17.657202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:17.657202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:17.657202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:17.657202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:17.657202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:17.657202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:17.657202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:17.657202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:17.657202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:17.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:17.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:17.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:17.664857 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:17.665056 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:17.665177 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:17.665350 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:17.665810 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:17.666034 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:17.666191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.666658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:17.666658 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:17.666658 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:17.666658 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:17.666658 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:17.666658 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:17.666658 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.666658 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:17.667361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.667361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:17.667361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:17.667361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:17.667361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.667361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:17.667361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:17.667361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.667361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:17.667361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:17.667361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:17.667361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.667771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:17.667771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:17.667771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:17.667771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:17.667771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:17.667771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:17.667771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:17.667771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:17.667771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:17.667771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:17.667771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:17.667771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:17.667771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:17.668176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.668176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:17.668382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.668382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:17.668382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:17.668382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:17.668382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:17.668382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:17.668382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.668382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:17.668382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:17.668382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.668382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:17.668382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:17.668382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.668382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:17.668382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:17.668382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:17.668732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:17.668732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:17.668732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.668732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:17.668732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:17.668732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:17.668732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:17.668732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:17.668732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:17.668732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:17.668732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:17.668732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:17.668732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:17.668732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:17.668732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:17.668732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:17.668732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:17.668732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:17.668732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:17.669055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:17.669055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:17.669055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:18.844993 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:18.845526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:18.845526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:18.846002 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:18.875166 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:18.875368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:18.876001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:18.876001 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:18.876001 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:18.876001 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:18.876001 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:18.876001 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:18.876001 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:18.876001 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:18.877010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:18.877010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:18.877010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:18.877010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:18.877010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:18.877010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:18.877010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:18.877010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:18.877010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:18.877010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:18.877010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:18.877010 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:18.877280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:18.877280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:18.877280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:18.877280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:18.877280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:18.877280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:18.877280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:18.877280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:18.877280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:18.877280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:18.877280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:18.877280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:18.877280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:18.877585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:18.877585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:18.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:18.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:18.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:18.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:18.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:18.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:18.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:18.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:18.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:18.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:18.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:18.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:18.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:18.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:18.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:18.878245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:18.878567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:18.878567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:18.878567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:18.878567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:18.878567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:18.878567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:18.878567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:18.878567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:18.878567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:18.878567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:18.878567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:18.878567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:18.878567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:18.878567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:18.878567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:18.878567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:18.878567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:18.878567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:18.878567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:18.878912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:18.878912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:18.878912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:22.643471 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:22.643893 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:22.643893 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:22.644534 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:22.644771 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:22.644921 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:22.645056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.645680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:22.645680 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:22.645680 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:22.645680 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:22.645680 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:22.645680 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:22.645680 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.645680 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:22.646428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.646428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:22.646428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:22.646428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:22.646428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.646428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:22.646428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:22.646428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.646428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:22.646428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:22.646428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:22.646428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.646764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:22.646764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:22.646764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:22.646764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:22.646764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:22.646764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:22.646764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:22.646764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:22.646764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:22.646764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:22.646764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:22.646764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:22.646764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:22.647073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.647073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:22.647335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.647335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:22.647335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:22.647335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:22.647335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:22.647335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:22.647335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.647335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:22.647335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:22.647335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.647335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:22.647335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:22.647335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.647335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:22.647335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:22.647335 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.647644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:22.647644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:22.647644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.647644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:22.647644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:22.647644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:22.647644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.647644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:22.647644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:22.647644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:22.647644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:22.647644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:22.647644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:22.647644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:22.647644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:22.647644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:22.647644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:22.647644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:22.647644 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:22.647968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:22.647968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.647968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:22.647968 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:22.648209 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:22.648354 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:22.648500 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:46:22.648865 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:22.649046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.649537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:22.649537 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:22.649537 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:22.649537 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:22.649537 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:22.649537 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:22.649537 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.649537 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:22.650127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.650127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:22.650127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:22.650127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:22.650127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.650127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:22.650127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:22.650127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:22.650127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.650127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:22.650127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:22.650127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:22.650127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:22.650127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:22.650127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:22.650458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:22.650458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:22.650458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:22.650458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:22.650458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:22.650458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:22.650458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:22.650458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.650458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:22.650889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.650889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:22.650889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:22.650889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:22.650889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.650889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:22.650889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:22.650889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:46:22.650889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:22.650889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:22.650889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.650889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:22.651101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:22.651101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:22.651101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.651101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:22.651101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:22.651101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:22.651101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:22.651101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:22.651101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:22.651101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:22.651101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:22.651101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:22.651101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:22.651101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:22.651414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:22.651414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:22.651414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.651414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:22.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:22.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:22.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:22.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:22.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:22.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:22.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:22.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:22.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:22.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:22.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:22.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.652153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:22.652153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:22.652153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:46:22.652153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:22.652153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:22.652153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.652153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:22.652153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:22.652153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:22.652153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.652153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:22.652153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:22.652153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:22.652153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:22.652153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:22.652153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:22.652153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:22.652153 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:22.652487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:22.652487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:22.652487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:22.652487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:22.652487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:22.652487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.652487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:22.653062 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:22.653298 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:22.653405 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:22.653545 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:22.653860 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:22.654066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.654532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:22.654532 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:22.654532 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:22.654532 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:22.654532 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:22.654532 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:22.654532 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.654532 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:22.655062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.655062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:22.655062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:22.655062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:22.655062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.655062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:22.655062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:22.655062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:22.655062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.655062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:22.655062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:22.655062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:22.655062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:22.655062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:22.655062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:22.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:22.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:22.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:22.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:22.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:22.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:22.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:22.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:22.655811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.655811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:22.655811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:22.655811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:22.655811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.655811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:22.655811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:22.655811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:22.655811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:22.655811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.655811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:22.656119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:22.656119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:22.656119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.656119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:22.656119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:22.656119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:22.656119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:22.656119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:22.656119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:22.656119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:22.656119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:22.656119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:22.656119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:22.656119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:22.656535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:22.656535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:22.656535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.656535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:22.656535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.656535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:22.656535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:22.656535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:22.656535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:22.656535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:22.656535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.656535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:22.656535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:22.656535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.656949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:22.656949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:22.656949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.656949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:22.656949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:22.656949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.656949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:22.656949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:22.656949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:22.656949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:22.656949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.656949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:22.656949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:22.656949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:22.656949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.656949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:22.656949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:22.656949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:22.656949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:22.657537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:22.657537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:22.657537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:22.657537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:22.657537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:22.657537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:22.657537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:22.657537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:22.657537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:22.657537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.657537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:22.665104 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:22.665337 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:22.665427 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:22.665558 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:22.665862 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:22.666063 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:22.666215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.666701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:22.666701 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:22.666701 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:22.666701 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:22.666701 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:22.666701 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:22.666701 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.666701 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:22.667303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.667303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:22.667303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:22.667303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:22.667303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.667303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:22.667303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:22.667303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.667303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:22.667303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:22.667303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:22.667303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.667664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:22.667664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:22.667664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:22.667664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:22.667664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:22.667664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:22.667664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:22.667664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:22.667664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:22.667664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:22.667664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:22.667664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:22.667664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:22.668042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.668042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:22.668042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.668042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:22.668042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:22.668042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:22.668042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:22.668042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:22.668042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.668042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:22.668042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:22.668042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.668042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:22.668042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:22.668042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.668467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:22.668467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:22.668467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:22.668467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:22.668467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:22.668467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.668467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:22.668467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:22.668467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:22.668467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:22.668467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:22.668467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:22.668467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:22.668467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:22.668467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:22.668467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:22.668467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:22.668467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:22.668467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:22.669023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:22.669023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:22.669023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:22.669023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:22.669023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:22.669023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:23.845452 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:23.846023 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:23.846023 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:23.846658 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:23.880001 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:23.880212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:23.880799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:23.880799 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:23.880799 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:23.880799 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:23.880799 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:23.880799 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:23.880799 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:23.880799 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:23.881634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:23.881634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:23.881634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:23.881634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:23.881634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:23.881634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:23.881634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:23.881634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:23.881634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:23.881634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:23.881634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:23.881634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:23.881935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:23.881935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:23.881935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:23.881935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:23.881935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:23.881935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:23.881935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:23.881935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:23.881935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:23.881935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:23.881935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:23.881935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:23.881935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:23.882233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:23.882233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:23.882424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:23.882424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:23.882424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:23.882424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:23.882424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:23.882424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:23.882424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:23.882424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:23.882424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:23.882424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:23.882424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:23.882424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:23.882424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:23.882424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:23.882424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:23.882424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:23.882916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:23.882916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:23.882916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:23.882916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:23.882916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:23.882916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:23.882916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:23.882916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:23.882916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:23.882916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:23.882916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:23.882916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:23.882916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:23.882916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:23.882916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:23.882916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:23.882916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:23.882916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:23.882916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:23.883352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:23.883352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:23.883352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:27.643861 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:27.644506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:27.644506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:27.644930 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:27.645070 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:27.645179 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:27.645322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.645713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:27.645713 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:27.645713 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:27.645713 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:27.645713 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:27.645713 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:27.645713 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.645713 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:27.646521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.646521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:27.646521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:27.646521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:27.646521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.646521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:27.646521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:27.646521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.646521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:27.646521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:27.646521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:27.646521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.646909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:27.646909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:27.646909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:27.646909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:27.646909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:27.646909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:27.646909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:27.646909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:27.646909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:27.646909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:27.646909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:27.646909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:27.646909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:27.647339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.647339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:27.647625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.647625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:27.647625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:27.647625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:27.647625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:27.647625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:27.647625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.647625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:27.647625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:27.647625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.647625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:27.647625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:27.647625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.647625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:27.647625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:27.647625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.647986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:27.647986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:27.647986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.647986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:27.647986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:27.647986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:27.647986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.647986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:27.647986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:27.647986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:27.647986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:27.647986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:27.647986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:27.647986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:27.647986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:27.647986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:27.647986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:27.647986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:27.647986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:27.648498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:27.648498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.648498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:27.648498 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:27.648498 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:27.648498 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:27.648748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:46:27.648917 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:27.649065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.649414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:27.649414 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:27.649414 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:27.649414 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:27.649414 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:27.649414 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:27.649414 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.649414 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:27.650137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.650137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:27.650137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:27.650137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:27.650137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.650137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:27.650137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:27.650137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:27.650137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.650137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:27.650137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:27.650137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:27.650137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:27.650137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:27.650137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:27.650486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:27.650486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:27.650486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:27.650486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:27.650486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:27.650486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:27.650486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:27.650486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.650486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:27.650933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.650933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:27.650933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:27.650933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:27.650933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.650933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:27.650933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:27.650933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:46:27.650933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:27.650933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:27.650933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.650933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:27.651224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:27.651224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:27.651224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.651224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:27.651224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:27.651224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:27.651224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:27.651224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:27.651224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:27.651224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:27.651224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:27.651224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:27.651224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:27.651224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:27.651467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:27.651467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:27.651467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.651467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:27.651601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.651601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:27.651601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:27.651601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:27.651601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:27.651601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:27.651601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.651601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:27.651601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:27.651601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.651601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:27.651601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:27.651601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.651601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:27.651601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:27.651601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:27.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:27.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:46:27.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:27.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:27.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:27.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:27.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:27.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:27.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:27.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:27.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:27.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:27.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:27.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:27.651871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:27.652165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:27.652165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:27.652165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:27.652165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:27.652165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:27.652165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.652165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:27.653324 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:27.653506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:27.653638 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:27.653766 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:27.654006 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:27.654207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.654611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:27.654611 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:27.654611 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:27.654611 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:27.654611 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:27.654611 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:27.654611 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.654611 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:27.655186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.655186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:27.655186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:27.655186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:27.655186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.655186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:27.655186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:27.655186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:27.655186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.655186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:27.655186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:27.655186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:27.655186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:27.655186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:27.655186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:27.655453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:27.655453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:27.655453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:27.655453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:27.655453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:27.655453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:27.655453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:27.655453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.655453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:27.655958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.655958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:27.655958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:27.655958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:27.655958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.655958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:27.655958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:27.655958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:27.655958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:27.655958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.655958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:27.656150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:27.656150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:27.656150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.656150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:27.656150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:27.656150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:27.656150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:27.656150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:27.656150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:27.656150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:27.656150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:27.656150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:27.656150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:27.656150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:27.656372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:27.656372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:27.656372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.656372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:27.656890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.656890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:27.656890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:27.656890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:27.656890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:27.656890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:27.656890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.656890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:27.656890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:27.656890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.656890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:27.656890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:27.656890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.656890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:27.656890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:27.656890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.656890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:27.657201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:27.657201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:27.657201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:27.657201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.657201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:27.657201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:27.657201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:27.657201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.657201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:27.657201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:27.657201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:27.657201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:27.657201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:27.657201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:27.657201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:27.657201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:27.657201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:27.657201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:27.657201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:27.657513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:27.657513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:27.657513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.657513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:27.665384 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:27.665573 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:27.665704 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:27.665856 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:27.666088 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:27.666306 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:27.666461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.666876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:27.666876 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:27.666876 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:27.666876 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:27.666876 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:27.666876 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:27.666876 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.666876 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:27.667543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.667543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:27.667543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:27.667543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:27.667543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.667543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:27.667543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:27.667543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.667543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:27.667543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:27.667543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:27.667543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.667782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:27.667782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:27.667782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:27.667782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:27.667782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:27.667782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:27.667782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:27.667782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:27.667782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:27.667782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:27.667782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:27.667782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:27.667782 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:27.667979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.667979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:27.668418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.668418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:27.668418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:27.668418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:27.668418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:27.668418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:27.668418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.668418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:27.668418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:27.668418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.668418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:27.668418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:27.668418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.668418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:27.668418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:27.668418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:27.668707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:27.668707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:27.668707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.668707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:27.668707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:27.668707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:27.668707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:27.668707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:27.668707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:27.668707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:27.668707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:27.668707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:27.668707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:27.668707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:27.668707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:27.668707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:27.668707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:27.668707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:27.668707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:27.668998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:27.668998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:27.668998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:28.845943 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:28.846588 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:28.846588 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:28.846798 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:28.874199 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:28.874414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:28.874906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:28.874906 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:28.874906 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:28.874906 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:28.874906 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:28.874906 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:28.874906 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:28.874906 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:28.875524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:28.875524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:28.875524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:28.875524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:28.875524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:28.875524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:28.875524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:28.875524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:28.875524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:28.875524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:28.875524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:28.875524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:28.875920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:28.875920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:28.875920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:28.875920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:28.875920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:28.875920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:28.875920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:28.875920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:28.875920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:28.875920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:28.875920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:28.875920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:28.875920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:28.876324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:28.876324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:28.876324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:28.876324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:28.876324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:28.876324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:28.876324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:28.876324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:28.876324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:28.876324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:28.876324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:28.876324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:28.876324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:28.876324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:28.876324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:28.876781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:28.876781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:28.876781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:28.876781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:28.876781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:28.876781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:28.876781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:28.876781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:28.876781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:28.876781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:28.876781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:28.876781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:28.876781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:28.876781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:28.876781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:28.876781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:28.876781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:28.876781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:28.876781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:28.877339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:28.877339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:28.877339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:28.877339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:28.877339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:28.877339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:32.644419 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:32.644909 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:32.645074 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:32.646003 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:32.646190 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:32.646354 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:32.646504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.647045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:32.647045 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:32.647045 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:32.647045 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:32.647045 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:32.647045 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:32.647045 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.647045 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:32.648223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.648223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:32.648223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:32.648223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:32.648223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.648223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:32.648223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:32.648223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.648223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:32.648223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:32.648223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:32.648223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.648685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:32.648685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:32.648685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:32.648685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:32.648685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:32.648685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:32.648685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:32.648685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:32.648685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:32.648685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:32.648685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:32.648685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:32.648685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:32.649028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.649028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:32.649651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.649651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:32.649651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:32.649651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:32.649651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:32.649651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:32.649651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.649651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:32.649651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:32.649651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.649651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:32.649651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:32.649651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.649651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:32.649651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:32.649651 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.650099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:32.650099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:32.650099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.650099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:32.650099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:32.650099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:32.650099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.650099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:32.650099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:32.650099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:32.650099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:32.650099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:32.650099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:32.650099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:32.650099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:32.650099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:32.650099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:32.650099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:32.650099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:32.650741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:32.650741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.650741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:32.650741 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:32.650741 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:32.650741 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:32.650741 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:46:32.650982 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:32.651095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.651562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:32.651562 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:32.651562 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:32.651562 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:32.651562 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:32.651562 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:32.651562 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.651562 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:32.652985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:32.652985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:32.652985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:32.652985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:32.652985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:32.652985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:32.652985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:32.652985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.652985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:32.653450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.653450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:32.653450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:32.653450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:32.653450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.653450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:32.653450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:32.653450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:46:32.653450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:32.653450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:32.653450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.653450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:32.653899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:32.653899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:32.653899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.653899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:32.653899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:32.653899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:32.653899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:32.653899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:32.653899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:32.653899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:32.653899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:32.653899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:32.653899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:32.653899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:32.654373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:32.654373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:32.654373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.654373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:32.654830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.654830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:32.654830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:32.654830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:32.654830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:32.654830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:32.654830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.654830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:32.654830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:32.654830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.654830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:32.654830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:32.654830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.654830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:32.654830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:32.654830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.655351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:32.655351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:32.655351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:46:32.655351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:32.655351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:32.655351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.655351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:32.655351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:32.655351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:32.655351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.655351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:32.655351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:32.655351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:32.655351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:32.655351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:32.655351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:32.655351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:32.655351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:32.655901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:32.655901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:32.655901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:32.655901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:32.655901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:32.655901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.655901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:32.655901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:32.655901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:32.655901 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:32.656229 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:32.656229 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:32.656372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.656744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:32.656744 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:32.656744 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:32.656744 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:32.656744 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:32.656744 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:32.656744 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.656744 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:32.657653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.657653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:32.657653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:32.657653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:32.657653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.657653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:32.657653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:32.657653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:32.657653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.657653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:32.657653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:32.657653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:32.657653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:32.657653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:32.657653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:32.658092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:32.658092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:32.658092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:32.658092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:32.658092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:32.658092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:32.658092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:32.658092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.658092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:32.658676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.658676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:32.658676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:32.658676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:32.658676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.658676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:32.658676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:32.658676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:32.658676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:32.658676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.658676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:32.659042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:32.659042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:32.659042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.659042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:32.659042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:32.659042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:32.659042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:32.659042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:32.659042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:32.659042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:32.659042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:32.659042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:32.659042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:32.659042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:32.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:32.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:32.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:32.659896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.659896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:32.659896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:32.659896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:32.659896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:32.659896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:32.659896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.659896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:32.659896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:32.659896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.659896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:32.659896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:32.659896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.659896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:32.659896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:32.659896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.659896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:32.660222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:32.660222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:32.660222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:32.660222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.660222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:32.660222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:32.660222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:32.660222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.660222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:32.660222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:32.660222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:32.660222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:32.660222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:32.660222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:32.660222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:32.660222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:32.660222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:32.660222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:32.660222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:32.660604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:32.660604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:32.660604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.660604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:32.665857 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:32.666273 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:32.666455 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:32.666697 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:32.667120 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:32.667361 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:32.667509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.668114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:32.668114 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:32.668114 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:32.668114 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:32.668114 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:32.668114 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:32.668114 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.668114 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:32.669168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.669168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:32.669168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:32.669168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:32.669168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.669168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:32.669168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:32.669168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.669168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:32.669168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:32.669168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:32.669168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.669608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:32.669608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:32.669608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:32.669608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:32.669608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:32.669608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:32.669608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:32.669608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:32.669608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:32.669608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:32.669608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:32.669608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:32.669608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:32.669960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.669960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:32.670403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.670403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:32.670403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:32.670403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:32.670403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:32.670403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:32.670403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.670403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:32.670403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:32.670403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.670403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:32.670403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:32.670403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.670403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:32.670403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:32.670403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:32.670915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:32.670915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:32.670915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.670915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:32.670915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:32.670915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:32.670915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:32.670915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:32.670915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:32.670915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:32.670915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:32.670915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:32.670915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:32.670915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:32.670915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:32.670915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:32.670915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:32.670915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:32.670915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:32.671420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:32.671420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:32.671420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:33.846301 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:33.846814 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:33.846867 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:33.847646 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:33.879732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:33.879916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:33.880344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:33.880344 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:33.880344 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:33.880344 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:33.880344 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:33.880344 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:33.880344 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:33.880344 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:33.880975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:33.880975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:33.880975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:33.880975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:33.880975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:33.880975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:33.880975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:33.880975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:33.880975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:33.880975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:33.880975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:33.880975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:33.881393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:33.881393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:33.881393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:33.881393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:33.881393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:33.881393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:33.881393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:33.881393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:33.881393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:33.881393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:33.881393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:33.881393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:33.881393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:33.881767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:33.881767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:33.881767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:33.881767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:33.881767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:33.881767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:33.881767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:33.881767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:33.881767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:33.881767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:33.881767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:33.881767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:33.881767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:33.881767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:33.881767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:33.882180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:33.882180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:33.882180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:33.882180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:33.882180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:33.882180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:33.882180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:33.882180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:33.882180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:33.882180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:33.882180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:33.882180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:33.882180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:33.882180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:33.882180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:33.882180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:33.882180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:33.882180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:33.882180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:33.882682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:33.882682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:33.882682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:33.882682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:33.882682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:33.882682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:37.644960 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:37.645508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:37.645508 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:37.645997 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:37.646184 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:37.646364 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:37.646519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.646988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:37.646988 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:37.646988 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:37.646988 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:37.646988 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:37.646988 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:37.646988 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.646988 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:37.647653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.647653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:37.647653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:37.647653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:37.647653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.647653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:37.647653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:37.647653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.647653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:37.647653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:37.647653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:37.647653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.647931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:37.647931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:37.647931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:37.647931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:37.647931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:37.647931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:37.647931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:37.647931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:37.647931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:37.647931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:37.647931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:37.647931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:37.647931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:37.648218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.648218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:37.648367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.648367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:37.648367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:37.648367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:37.648367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:37.648367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:37.648367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.648367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:37.648367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:37.648367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.648367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:37.648367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:37.648367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.648367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:37.648367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:37.648367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.648802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:37.648802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:37.648802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.648802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:37.648802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:37.648802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:37.648802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.648802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:37.648802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:37.648802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:37.648802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:37.648802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:37.648802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:37.648802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:37.648802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:37.648802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:37.648802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:37.648802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:37.648802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:37.649248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:37.649248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.649248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:37.650208 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:37.650366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:37.650470 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:37.650559 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:46:37.650865 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:37.651038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.651452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:37.651452 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:37.651452 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:37.651452 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:37.651452 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:37.651452 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:37.651452 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.651452 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:37.651973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.651973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:37.651973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:37.651973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:37.651973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.651973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:37.651973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:37.651973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:37.651973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.651973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:37.651973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:37.651973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:37.651973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:37.651973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:37.651973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:37.652276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:37.652276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:37.652276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:37.652276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:37.652276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:37.652276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:37.652276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:37.652276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.652276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:37.652536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.652536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:37.652536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:37.652536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:37.652536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.652536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:37.652536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:37.652536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:46:37.652536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:37.652536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:37.652536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.652536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:37.652788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:37.652788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:37.652788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.652788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:37.652788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:37.652788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:37.652788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:37.652788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:37.652788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:37.652788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:37.652788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:37.652788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:37.652788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:37.652788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:37.653030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:37.653030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:37.653030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.653030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:37.653322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.653322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:37.653322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:37.653322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:37.653322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:37.653322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:37.653322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.653322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:37.653322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:37.653322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.653322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:37.653322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:37.653322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.653322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:37.653322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:37.653322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.653710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:37.653710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:37.653710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:46:37.653710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:37.653710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:37.653710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.653710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:37.653710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:37.653710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:37.653710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.653710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:37.653710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:37.653710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:37.653710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:37.653710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:37.653710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:37.653710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:37.653710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:37.654031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:37.654031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:37.654031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:37.654031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:37.654031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:37.654031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.654031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:37.655445 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:37.655656 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:37.655778 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:37.655925 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:37.656244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:37.656521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.656993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:37.656993 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:37.656993 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:37.656993 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:37.656993 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:37.656993 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:37.656993 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.656993 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:37.657512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.657512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:37.657512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:37.657512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:37.657512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.657512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:37.657512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:37.657512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:37.657512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.657512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:37.657512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:37.657512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:37.657512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:37.657512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:37.657512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:37.657973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:37.657973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:37.657973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:37.657973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:37.657973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:37.657973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:37.657973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:37.657973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.657973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:37.658264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.658264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:37.658264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:37.658264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:37.658264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.658264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:37.658264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:37.658264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:37.658264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:37.658264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.658264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:37.658622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:37.658622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:37.658622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.658622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:37.658622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:37.658622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:37.658622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:37.658622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:37.658622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:37.658622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:37.658622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:37.658622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:37.658622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:37.658622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:37.659038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:37.659038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:37.659038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.659038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:37.659038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.659038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:37.659038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:37.659038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:37.659038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:37.659038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:37.659038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.659038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:37.659038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:37.659038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.659663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:37.659663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:37.659663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.659663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:37.659663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:37.659663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.659663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:37.659663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:37.659663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:37.659663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:37.659663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.659663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:37.659663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:37.659663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:37.659663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.659663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:37.659663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:37.659663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:37.659663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:37.660206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:37.660206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:37.660206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:37.660206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:37.660206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:37.660206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:37.660206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:37.660206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:37.660206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:37.660206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.660206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:37.666224 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:37.666406 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:37.666512 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:37.666653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:37.666961 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:37.667177 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:37.667334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.667769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:37.667769 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:37.667769 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:37.667769 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:37.667769 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:37.667769 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:37.667769 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.667769 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:37.668400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.668400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:37.668400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:37.668400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:37.668400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.668400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:37.668400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:37.668400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.668400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:37.668400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:37.668400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:37.668400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.668673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:37.668673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:37.668673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:37.668673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:37.668673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:37.668673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:37.668673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:37.668673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:37.668673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:37.668673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:37.668673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:37.668673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:37.668673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:37.668903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.668903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:37.669125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.669125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:37.669125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:37.669125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:37.669125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:37.669125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:37.669125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.669125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:37.669125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:37.669125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.669125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:37.669125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:37.669125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.669125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:37.669125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:37.669125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:37.669717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:37.669717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:37.669717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.669717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:37.669717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:37.669717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:37.669717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:37.669717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:37.669717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:37.669717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:37.669717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:37.669717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:37.669717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:37.669717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:37.669717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:37.669717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:37.669717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:37.669717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:37.669717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:37.670336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:37.670336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:37.670336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:38.846809 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:38.849335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:38.849335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:38.849335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:38.882786 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:38.882786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:38.883438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:38.883438 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:38.883438 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:38.883438 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:38.883438 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:38.883438 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:38.883438 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:38.883438 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:38.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:38.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:38.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:38.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:38.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:38.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:38.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:38.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:38.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:38.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:38.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:38.884323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:38.884838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:38.884838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:38.884838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:38.884838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:38.884838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:38.884838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:38.884838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:38.884838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:38.884838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:38.884838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:38.884838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:38.884838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:38.884838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:38.885200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:38.885200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:38.885200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:38.885200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:38.885200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:38.885200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:38.885200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:38.885200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:38.885200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:38.885200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:38.885200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:38.885200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:38.885200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:38.885200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:38.885200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:38.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:38.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:38.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:38.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:38.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:38.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:38.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:38.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:38.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:38.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:38.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:38.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:38.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:38.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:38.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:38.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:38.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:38.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:38.885699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:38.886234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:38.886234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:38.886234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:38.886234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:38.886234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:38.886234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:42.645365 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:42.646791 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:42.646791 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:42.647091 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:42.647290 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:42.647446 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:42.647605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.649690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:42.649690 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:42.649690 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:42.649690 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:42.649690 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:42.649690 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:42.649690 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.649690 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:42.649690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.649690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:42.649690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:42.649690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:42.649690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.649690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:42.649690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:42.650154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.650154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:42.650154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:42.650154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:42.650154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.650154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:42.650154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:42.650154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:42.650154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:42.650154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:42.650154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:42.650154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:42.650154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:42.650154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:42.650516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:42.650516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:42.650516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:42.650516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:42.650516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.650516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:42.650516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.650516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:42.650516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:42.650516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:42.650516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:42.650516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:42.650516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.650516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:42.650824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:42.651181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:42.651181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:42.651181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:42.651181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:42.651181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:42.651181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:42.651181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:42.651181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:42.651181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.651181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:42.651713 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:42.652168 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:42.652168 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:42.652168 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:46:42.652366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:42.652560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.654701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:42.654701 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:42.654701 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:42.654701 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:42.654701 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:42.654701 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:42.654701 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.654701 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:42.654701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.654701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:42.654701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:42.654701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:42.654701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.654701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:42.654701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:42.654701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:42.654701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.654701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:42.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:42.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:42.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:42.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:42.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:42.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:42.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:42.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:42.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:42.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:42.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:42.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:42.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:42.655083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.655376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:42.655376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:42.655376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:42.655376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.655376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:42.655376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:42.655376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:46:42.655376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:42.655376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:42.655376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.655376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:42.655376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:42.655376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:42.655579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.655579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:42.655579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:42.655579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:42.655579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:42.655579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:42.655579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:42.655579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:42.655579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:42.655579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:42.655579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:42.655579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:42.655579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:42.655579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:42.655818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.655818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:42.655818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.655818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:42.655818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:42.655818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:42.655818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:42.655818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:42.655818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.655818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:42.655818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:42.655818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.655818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:42.655818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:42.655818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.656109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:42.656109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:42.656109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.656109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:42.656109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:42.656109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:46:42.656109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:42.656109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:42.656109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.656109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:42.656109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:42.656109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:42.656109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.656109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:42.656109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:42.656109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:42.656109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:42.656109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:42.656109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:42.656482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:42.656482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:42.656482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:42.656482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:42.656482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:42.656482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:42.656482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:42.656482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.656482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:42.656995 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:42.657496 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:42.657496 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:42.657496 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:42.657745 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:42.657966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.660218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:42.660218 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:42.660218 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:42.660218 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:42.660218 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:42.660218 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:42.660218 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.660218 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:42.660218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.660218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:42.660218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:42.660218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:42.660218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.660218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:42.660218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:42.660218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:42.660218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.660218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:42.660582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:42.660582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:42.660582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:42.660582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:42.660582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:42.660582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:42.660582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:42.660582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:42.660582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:42.660582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:42.660582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:42.660582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:42.660582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.660582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:42.660582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.660892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:42.660892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:42.660892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:42.660892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.660892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:42.660892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:42.660892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:42.660892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:42.660892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.660892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:42.660892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:42.660892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:42.660892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.661108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:42.661108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:42.661108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:42.661108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:42.661108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:42.661108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:42.661108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:42.661108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:42.661108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:42.661108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:42.661108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:42.661108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:42.661108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:42.661349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.661349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:42.661349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.661349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:42.661349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:42.661349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:42.661349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:42.661349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:42.661349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.661349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:42.661349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:42.661349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.661349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:42.661349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:42.661349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.661600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:42.661600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:42.661600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.661600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:42.661600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:42.661600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:42.661600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:42.661600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.661600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:42.661600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:42.661600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:42.661600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.661600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:42.661600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:42.661600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:42.661600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:42.661600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:42.661600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:42.661600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:42.661911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:42.661911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:42.661911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:42.661911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:42.661911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:42.661911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:42.661911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.661911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:42.666421 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:42.666672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:42.666841 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:42.667243 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:42.667618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:42.667823 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:42.667982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.669617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:42.669617 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:42.669617 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:42.669617 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:42.669617 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:42.669617 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:42.669617 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.669617 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:42.669617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.669617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:42.669617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:42.669617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:42.669617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.669617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:42.669617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:42.670098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.670098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:42.670098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:42.670098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:42.670098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.670098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:42.670098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:42.670098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:42.670098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:42.670098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:42.670098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:42.670098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:42.670098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:42.670098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:42.670525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:42.670525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:42.670525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:42.670525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:42.670525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.670525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:42.670525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.670525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:42.670525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:42.670525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:42.670525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:42.670525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:42.670525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.670525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:42.670960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:42.671562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:42.671562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:42.671562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:42.671562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:42.671562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:42.671562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:42.671562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:42.671562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:42.671562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:42.671562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:43.847873 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:43.848428 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:43.848428 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:43.849232 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:43.889464 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:43.890877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:43.890877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:43.890877 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:43.890877 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:43.890877 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:43.890877 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:43.890877 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:43.890877 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:43.890877 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:43.891786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:43.891786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:43.891786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:43.891786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:43.891786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:43.891786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:43.891786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:43.891786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:43.891786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:43.891786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:43.891786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:43.891786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:43.892064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:43.892064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:43.892064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:43.892064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:43.892064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:43.892064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:43.892064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:43.892064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:43.892064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:43.892064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:43.892064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:43.892064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:43.892064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:43.892438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:43.892438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:43.893037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:43.893037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:43.893037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:43.893037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:43.893037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:43.893037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:43.893037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:43.893037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:43.893037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:43.893037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:43.893037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:43.893037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:43.893037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:43.893037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:43.893037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:43.893037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:43.893358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:43.893358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:43.893358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:43.893358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:43.893358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:43.893358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:43.893358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:43.893358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:43.893358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:43.893358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:43.893358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:43.893358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:43.893358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:43.893358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:43.893358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:43.893358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:43.893358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:43.893358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:43.893358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:43.893672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:43.893672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:43.893672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:47.646250 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:47.646864 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:47.646864 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:47.647380 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:47.647603 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:47.647736 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:47.647868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.648344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:47.648344 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:47.648344 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:47.648344 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:47.648344 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:47.648344 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:47.648344 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.648344 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:47.649047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.649047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:47.649047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:47.649047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:47.649047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.649047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:47.649047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:47.649047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.649047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:47.649047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:47.649047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:47.649047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.649480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:47.649480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:47.649480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:47.649480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:47.649480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:47.649480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:47.649480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:47.649480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:47.649480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:47.649480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:47.649480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:47.649480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:47.649480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:47.649869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.649869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:47.649869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.649869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:47.649869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:47.649869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:47.649869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:47.649869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:47.649869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.649869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:47.649869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:47.649869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.649869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:47.649869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:47.649869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.650367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:47.650367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:47.650367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.650367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:47.650367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:47.650367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.650367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:47.650367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:47.650367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:47.650367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.650367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:47.650367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:47.650367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:47.650367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:47.650367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:47.650367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:47.650367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:47.650367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:47.650367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:47.650920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:47.650920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:47.650920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:47.650920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:47.650920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.650920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:47.651933 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:47.652106 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:47.652214 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:47.652331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:46:47.652645 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:47.652792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.653198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:47.653198 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:47.653198 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:47.653198 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:47.653198 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:47.653198 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:47.653198 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.653198 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:47.653854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.653854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:47.653854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:47.653854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:47.653854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.653854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:47.653854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:47.653854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:47.653854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.653854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:47.653854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:47.653854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:47.653854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:47.653854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:47.653854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:47.654331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:47.654331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:47.654331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:47.654331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:47.654331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:47.654331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:47.654331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:47.654331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.654331 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:47.654625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.654625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:47.654625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:47.654625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:47.654625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.654625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:47.654625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:47.654625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:46:47.654625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:47.654625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:47.654625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.654625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:47.654945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:47.654945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:47.654945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.654945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:47.654945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:47.654945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:47.654945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:47.654945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:47.654945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:47.654945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:47.654945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:47.654945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:47.654945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:47.654945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:47.655312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:47.655312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:47.655312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.655312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:47.655420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.655420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:47.655420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:47.655420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:47.655420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:47.655420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:47.655420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.655420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:47.655420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:47.655420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.655420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:47.655420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:47.655420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.655420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:47.655420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:47.655420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.655863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:47.655863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:47.655863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:46:47.655863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:47.655863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:47.655863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.655863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:47.655863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:47.655863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:47.655863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.655863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:47.655863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:47.655863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:47.655863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:47.655863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:47.655863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:47.655863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:47.655863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:47.656349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:47.656349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:47.656349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:47.656349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:47.656349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:47.656349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.656349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:47.657771 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:47.657771 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:47.657771 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:47.658082 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:47.658082 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:47.658310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.658693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:47.658693 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:47.658693 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:47.658693 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:47.658693 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:47.658693 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:47.658693 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.658693 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:47.659217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.659217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:47.659217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:47.659217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:47.659217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.659217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:47.659217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:47.659217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:47.659217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.659217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:47.659217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:47.659217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:47.659217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:47.659217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:47.659217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:47.660300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:47.660300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:47.660300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:47.660300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:47.660300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:47.660300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:47.660300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:47.660300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.660300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:47.660300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.660300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:47.660300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:47.660300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:47.660300 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.661448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:47.661448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:47.661448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:47.661448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:47.661448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.661448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:47.661448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:47.661448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:47.661448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.661448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:47.661448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:47.661448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:47.661448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:47.661945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:47.661945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:47.661945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:47.661945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:47.661945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:47.661945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:47.661945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:47.661945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:47.661945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:47.661945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.661945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:47.661945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.662281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:47.662281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:47.662281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:47.662281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:47.662281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:47.662281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.662281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:47.662281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:47.662281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.662281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:47.662281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:47.662281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.662281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:47.662281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:47.662281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.662281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:47.662281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:47.662281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:47.662281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.662789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:47.666815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:47.667042 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:47.667137 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:47.667294 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:47.667637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:47.667815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:47.667936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:47.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:47.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:47.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:47.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:47.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:47.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:47.669097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.669097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:47.669097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:47.669097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:47.669097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.669097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:47.669097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:47.669097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.669097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:47.669097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:47.669097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:47.669097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.669855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:47.669855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:47.669855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:47.669855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:47.669855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:47.669855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:47.669855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:47.669855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:47.669855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:47.669855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:47.669855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:47.669855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:47.669855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:47.670168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.670168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:47.670168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.670168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:47.670168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:47.670168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:47.670168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:47.670168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:47.670168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.670168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:47.670168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:47.670168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.670168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:47.670168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:47.670168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.670498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:47.670498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:47.670498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:47.670498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:47.670498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:47.670498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.670498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:47.670498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:47.670498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:47.670498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:47.670498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:47.670498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:47.670498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:47.670498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:47.670498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:47.670498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:47.670498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:47.670498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:47.670498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:47.670799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:47.670799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:47.670799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:47.670799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:47.670799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:47.670799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:48.848348 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:48.848926 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:48.848926 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:48.849584 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:48.876392 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:48.876685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:48.877338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:48.877338 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:48.877338 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:48.877338 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:48.877338 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:48.877338 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:48.877338 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:48.877338 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:48.878073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:48.878073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:48.878073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:48.878073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:48.878073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:48.878073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:48.878073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:48.878073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:48.878073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:48.878073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:48.878073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:48.878073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:48.878350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:48.878350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:48.878350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:48.878350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:48.878350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:48.878350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:48.878350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:48.878350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:48.878350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:48.878350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:48.878350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:48.878350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:48.878350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:48.878678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:48.878678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:48.878988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:48.878988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:48.878988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:48.878988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:48.878988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:48.878988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:48.878988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:48.878988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:48.878988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:48.878988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:48.878988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:48.878988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:48.878988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:48.878988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:48.878988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:48.878988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:48.879812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:48.879812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:48.879812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:48.879812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:48.879812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:48.879812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:48.879812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:48.879812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:48.879812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:48.879812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:48.879812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:48.879812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:48.879812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:48.879812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:48.879812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:48.879812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:48.879812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:48.879812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:48.879812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:48.880285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:48.880285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:48.880285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:52.646665 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:52.647296 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:52.647417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:52.648112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:52.648349 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:52.648497 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:52.648645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.649241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:52.649241 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:52.649241 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:52.649241 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:52.649241 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:52.649241 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:52.649241 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.649241 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:52.650087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.650087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:52.650087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:52.650087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:52.650087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.650087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:52.650087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:52.650087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.650087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:52.650087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:52.650087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:52.650087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.650317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:52.650317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:52.650317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:52.650317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:52.650317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:52.650317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:52.650317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:52.650317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:52.650317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:52.650317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:52.650317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:52.650317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:52.650317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:52.650559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.650559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:52.650866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.650866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:52.650866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:52.650866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:52.650866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:52.650866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:52.650866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.650866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:52.650866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:52.650866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.650866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:52.650866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:52.650866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.650866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:52.650866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:52.650866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.651155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:52.651155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:52.651155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.651155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:52.651155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:52.651155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:52.651155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.651155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:52.651155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:52.651155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:52.651155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:52.651155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:52.651155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:52.651155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:52.651155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:52.651155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:52.651155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:52.651155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:52.651155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:52.651473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:52.651473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.651473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:52.652224 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:52.652509 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:52.652635 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:52.652735 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:46:52.653042 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:52.653232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.653721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:52.653721 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:52.653721 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:52.653721 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:52.653721 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:52.653721 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:52.653721 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.653721 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:52.654311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.654311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:52.654311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:52.654311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:52.654311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.654311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:52.654311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:52.654311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:52.654311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.654311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:52.654311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:52.654311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:52.654311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:52.654311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:52.654311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:52.654584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:52.654584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:52.654584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:52.654584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:52.654584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:52.654584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:52.654584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:52.654584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.654584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:52.655048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.655048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:52.655048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:52.655048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:52.655048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.655048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:52.655048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:52.655048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:46:52.655048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:52.655048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:52.655048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.655048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:52.655268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:52.655268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:52.655268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.655268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:52.655268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:52.655268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:52.655268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:52.655268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:52.655268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:52.655268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:52.655268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:52.655268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:52.655268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:52.655268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:52.655574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:52.655574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:52.655574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.655574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:52.655943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.655943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:52.655943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:52.655943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:52.655943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:52.655943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:52.655943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.655943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:52.655943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:52.655943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.655943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:52.655943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:52.655943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.655943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:52.655943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:52.655943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.656756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:52.656756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:52.656756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:46:52.656756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:52.656756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:52.656756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.656756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:52.656756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:52.656756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:52.656756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.656756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:52.656756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:52.656756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:52.656756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:52.656756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:52.656756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:52.656756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:52.656756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:52.657188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:52.657188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:52.657188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:52.657188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:52.657188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:52.657188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.657188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:52.657733 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:52.657977 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:52.658072 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:52.658169 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:52.658524 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:52.658734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.659131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:52.659131 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:52.659131 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:52.659131 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:52.659131 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:52.659131 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:52.659131 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.659131 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:52.659762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.659762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:52.659762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:52.659762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:52.659762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.659762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:52.659762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:52.659762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:52.659762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.659762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:52.659762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:52.659762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:52.659762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:52.659762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:52.659762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:52.660028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:52.660028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:52.660028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:52.660028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:52.660028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:52.660028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:52.660028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:52.660028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.660028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:52.660479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.660479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:52.660479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:52.660479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:52.660479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.660479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:52.660479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:52.660479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:52.660479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:52.660479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.660479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:52.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:52.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:52.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:52.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:52.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:52.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:52.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:52.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:52.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:52.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:52.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:52.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:52.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:52.661286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:52.661286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:52.661286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.661286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:52.661459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.661459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:52.661459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:52.661459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:52.661459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:52.661459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:52.661459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.661459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:52.661459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:52.661459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.661459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:52.661459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:52.661459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.661459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:52.661459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:52.661459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.661459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:52.661755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:52.661755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:52.661755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:52.661755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.661755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:52.661755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:52.661755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:52.661755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.661755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:52.661755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:52.661755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:52.661755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:52.661755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:52.661755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:52.661755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:52.661755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:52.661755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:52.661755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:52.661755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:52.662060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:52.662060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:52.662060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.662060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:52.667155 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:52.667515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:52.667638 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:52.667805 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:52.668092 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:52.668309 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:52.668457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.668904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:52.668904 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:52.668904 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:52.668904 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:52.668904 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:52.668904 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:52.668904 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.668904 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:52.669729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.669729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:52.669729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:52.669729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:52.669729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.669729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:52.669729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:52.669729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.669729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:52.669729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:52.669729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:52.669729 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.669954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:52.669954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:52.669954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:52.669954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:52.669954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:52.669954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:52.669954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:52.669954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:52.669954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:52.669954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:52.669954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:52.669954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:52.669954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:52.670237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.670237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:52.670903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.670903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:52.670903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:52.670903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:52.670903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:52.670903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:52.670903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.670903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:52.670903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:52.670903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.670903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:52.670903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:52.670903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.670903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:52.670903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:52.670903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:52.671180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:52.671180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:52.671180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.671180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:52.671180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:52.671180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:52.671180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:52.671180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:52.671180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:52.671180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:52.671180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:52.671180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:52.671180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:52.671180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:52.671180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:52.671180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:52.671180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:52.671180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:52.671180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:52.671512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:52.671512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:52.671512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:53.848838 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:53.851321 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:53.851321 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:53.851321 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:53.855276 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=551,neutron:DELETE=5 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:46:53.886867 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:53.887117 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:53.889903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:53.889903 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:53.889903 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:53.889903 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:53.889903 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:53.889903 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:53.889903 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:53.889903 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:53.889903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:53.889903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:53.889903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:53.889903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:53.889903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:53.889903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:53.889903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:53.890482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:53.890482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:53.890482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:53.890482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:53.890482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:53.890482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:53.890482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:53.890482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:53.890482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:53.890482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:53.890482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:53.890482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:53.890482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:53.890482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:53.890958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:53.890958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:53.890958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:53.890958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:53.890958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:53.890958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:53.890958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:53.890958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:53.890958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:53.890958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:53.890958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:53.890958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:53.890958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:53.890958 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:53.891398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:53.891941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:53.891941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:53.891941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:53.891941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:53.891941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:53.891941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:53.891941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:53.891941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:53.891941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:57.647346 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:57.648050 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:57.648050 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:57.648636 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:57.648842 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:57.648985 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:57.649121 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.649749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:57.649749 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:57.649749 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:57.649749 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:57.649749 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:57.649749 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:57.649749 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.649749 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:57.650519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.650519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:57.650519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:57.650519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:57.650519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.650519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:57.650519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:57.650519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.650519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:57.650519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:57.650519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:57.650519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.651059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:57.651059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:57.651059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:57.651059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:57.651059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:57.651059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:57.651059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:57.651059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:57.651059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:57.651059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:57.651059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:57.651059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:57.651059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:57.651432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.651432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:57.651432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.651432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:57.651432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:57.651432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:57.651432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:57.651432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:57.651432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.651432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:57.651432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:57.651432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.651432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:57.651432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:57.651432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.651743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:57.651743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:57.651743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.651743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:46:57.651743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:57.651743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.651743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:57.651743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:57.651743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:57.651743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.651743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:57.651743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:57.651743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:57.651743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:57.651743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:57.651743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:57.651743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:57.651743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:57.651743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:57.652154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:57.652154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:57.652154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:57.652154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:57.652154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.652154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:57.652842 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:57.653089 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:57.653178 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:57.653289 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:46:57.653651 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:57.653836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.654341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:57.654341 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:57.654341 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:57.654341 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:57.654341 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:57.654341 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:57.654341 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.654341 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:57.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:57.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:57.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:57.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:57.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:57.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:57.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:57.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:57.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:57.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:57.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:57.654915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:57.655346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:57.655346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:57.655346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:57.655346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:57.655346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:57.655346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:57.655346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:57.655346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.655346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:57.655619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.655619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:57.655619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:57.655619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:57.655619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.655619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:57.655619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:57.655619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:46:57.655619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:57.655619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:57.655619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.655619 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:57.655841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:57.655841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:57.655841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.655841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:57.655841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:57.655841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:57.655841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:57.655841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:57.655841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:57.655841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:57.655841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:57.655841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:57.655841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:57.655841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:57.656154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:57.656154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:57.656154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.656154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:57.656290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.656290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:57.656290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:57.656290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:57.656290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:57.656290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:57.656290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.656290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:57.656290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:57.656290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.656290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:57.656290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:57.656290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.656290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:57.656290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:57.656290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.656688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:46:57.656688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:46:57.656688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:46:57.656688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:57.656688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:57.656688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.656688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:57.656688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:57.656688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:57.656688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.656688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:57.656688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:57.656688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:57.656688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:57.656688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:57.656688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:57.656688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:57.656688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:57.657006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:57.657006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:57.657006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:57.657006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:57.657006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:57.657006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.657006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:57.658177 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:57.658470 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:57.658563 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:57.658680 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:57.659116 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:57.659401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.659959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:57.659959 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:57.659959 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:57.659959 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:57.659959 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:57.659959 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:57.659959 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.659959 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:57.660508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.660508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:46:57.660508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:57.660508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:57.660508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.660508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:57.660508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:57.660508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:46:57.660508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.660508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:46:57.660508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:57.660508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:46:57.660508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:57.660508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:46:57.660508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:46:57.660952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:57.660952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:46:57.660952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:57.660952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:46:57.660952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:46:57.660952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:57.660952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:46:57.660952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.660952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:46:57.661280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.661280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:57.661280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:57.661280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:57.661280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.661280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:57.661280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:57.661280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:57.661280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:57.661280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.661280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:57.661690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:57.661690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:57.661690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.661690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:57.661690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:57.661690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:57.661690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:57.661690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:57.661690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:57.661690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:57.661690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:57.661690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:57.661690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:57.661690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:57.662060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:57.662060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:57.662060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.662060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:57.662060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.662060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:57.662060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:57.662060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:57.662060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:57.662060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:57.662060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.662060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:57.662060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:57.662060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.662496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:57.662496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:57.662496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.662496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:57.662496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:57.662496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.662496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:46:57.662496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:46:57.662496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:46:57.662496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:46:57.662496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.662496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:57.662496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:57.662496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:57.662496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.662496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:57.662496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:57.662496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:57.662496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:57.663074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:57.663074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:57.663074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:57.663074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:57.663074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:57.663074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:57.663074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:57.663074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:57.663074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:57.663074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.663074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:57.667739 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:57.668087 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:57.668175 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:57.668370 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:57.668871 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:57.669117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:57.669276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.669878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:57.669878 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:57.669878 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:57.669878 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:57.669878 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:57.669878 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:57.669878 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.669878 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:57.670581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.670581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:57.670581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:57.670581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:57.670581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.670581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:57.670581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:57.670581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.670581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:57.670581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:57.670581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:57.670581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.671050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:57.671050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:57.671050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:57.671050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:57.671050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:57.671050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:57.671050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:57.671050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:57.671050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:57.671050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:57.671050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:57.671050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:57.671050 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:57.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:57.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:57.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:57.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:57.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:57.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:57.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:57.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:57.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:57.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:57.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.671925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:57.671925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:57.671925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:57.671925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:46:57.671925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:57.671925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.671925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:57.671925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:57.671925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:57.671925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:57.671925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:57.671925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:57.671925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:57.671925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:57.671925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:57.671925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:57.671925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:57.671925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:57.671925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:57.672462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:57.672462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:57.672462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:57.672462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:57.672462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:57.672462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:46:58.849940 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:46:58.850567 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:46:58.850567 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:46:58.851077 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:46:58.883322 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:46:58.883322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:58.883531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:46:58.883531 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:58.883531 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:46:58.883531 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:46:58.883531 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:58.883531 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:46:58.883531 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:58.883531 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:46:58.884189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:58.884189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:46:58.884189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:58.884189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:46:58.884189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:58.884189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:58.884189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:58.884189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:58.884189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:58.884189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:58.884189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:46:58.884189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:58.884664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:46:58.884664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:58.884664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:46:58.884664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:58.884664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:46:58.884664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:46:58.884664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:58.884664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:46:58.884664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:58.884664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:46:58.884664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:46:58.884664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:58.884664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:46:58.885007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:58.885007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:46:58.885303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:58.885303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:46:58.885303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:46:58.885303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:46:58.885303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:46:58.885303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:46:58.885303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:58.885303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:46:58.885303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:46:58.885303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:58.885303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:46:58.885303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:46:58.885303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:46:58.885303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:46:58.885303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:46:58.885303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:46:58.885654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:46:58.885654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:46:58.885654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:58.885654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:58.885654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:46:58.885654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:46:58.885654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:46:58.885654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:46:58.885654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:46:58.885654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:46:58.885654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:46:58.885654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:46:58.885654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:46:58.885654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:46:58.885654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:46:58.885654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:46:58.885654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:46:58.885654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:46:58.885654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:46:58.885936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:46:58.885936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:46:58.885936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:02.647786 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:02.648190 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:02.648190 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:02.649269 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:02.649269 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:02.649269 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:02.649269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:02.649470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:02.649470 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:02.649470 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:02.649470 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:02.649470 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:02.649470 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:02.649470 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:02.649470 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:02.650093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:02.650093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:02.650093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:02.650093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:02.650093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:02.650093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:02.650093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:02.650093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:02.650093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:02.650093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:02.650093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:02.650093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:02.650535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:02.650535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:02.650535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:02.650535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:02.650535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:02.650535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:02.650535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:02.650535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:02.650535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:02.650535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:02.650535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:02.650535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:02.650535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:02.650990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:02.650990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:02.650990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:02.650990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:02.650990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:02.650990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:02.650990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:02.650990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:02.650990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:02.650990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:02.650990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:02.650990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:02.650990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:02.650990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:02.650990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:02.651485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:02.651485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:02.651485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:02.651485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:02.651485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:02.651485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:02.651485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:02.651485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:02.651485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:02.651485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:02.651485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:02.651485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:02.651485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:02.651485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:02.651485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:02.651485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:02.651485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:02.651485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:02.651485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:02.652089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:02.652089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:02.652089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:02.652089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:02.652089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:02.652089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:02.653119 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:02.653303 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:02.653382 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:02.653471 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:47:02.654071 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:02.654071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:02.654184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:02.654184 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:02.654184 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:02.654184 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:02.654184 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:02.654184 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:02.654184 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:02.654184 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:02.654724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:02.654724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:02.654724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:02.654724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:02.654724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:02.654724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:02.654724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:02.654724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:02.654724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:02.654724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:02.654724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:02.654724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:02.654724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:02.654724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:02.654724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:02.655204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:02.655204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:02.655204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:02.655204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:02.655204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:02.655204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:02.655204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:02.655204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:02.655204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:02.655518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:02.655518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:02.655518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:02.655518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:02.655518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:02.655518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:02.655518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:02.655518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:47:02.655518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:02.655518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:02.655518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:02.655518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:02.655810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:02.655810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:02.655810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:02.655810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:02.655810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:02.655810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:02.655810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:02.655810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:02.655810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:02.655810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:02.655810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:02.655810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:02.655810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:02.655810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:02.656022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:02.656022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:02.656022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:02.656022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:02.656022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:02.656022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:02.656022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:02.656022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:02.656022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:02.656022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:02.656022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:02.656022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:02.656022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:02.656022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:02.656239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:02.656239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:02.656239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:02.656239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:02.656239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:02.656239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:02.656239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:02.656239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:02.656239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:47:02.656239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:02.656239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:02.656239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:02.656239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:02.656239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:02.656239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:02.656239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:02.656239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:02.656239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:02.656239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:03.122506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:03.122506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:03.122506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:03.122506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:03.122506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:03.122506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:03.122506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:03.122506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:03.122506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:03.122506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:03.122506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.122506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:03.122506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:03.122506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:03.123683 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:03.123683 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:03.123683 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:03.123683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.123683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:03.123683 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:03.123683 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:03.123683 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:03.123683 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:03.123683 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:03.123683 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.123683 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:03.124353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.124353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:03.124353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:03.124353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:03.124353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:03.124353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:03.124353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:03.124353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:03.124353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:03.124353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:03.124353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:03.124353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:03.124353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:03.124353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:03.124353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:03.125101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:03.125101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:03.125101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:03.125101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:03.125101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:03.125101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:03.125101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:03.125101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.125101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:03.125101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.125101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:03.125101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:03.125101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:03.125101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:03.125786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:03.125786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:03.125786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:03.125786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:03.125786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:03.125786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:03.125786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:03.125786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:03.125786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:03.125786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:03.125786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:03.125786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:03.125786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:03.126304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:03.126304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:03.126304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:03.126304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:03.126304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:03.126304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:03.126304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:03.126304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:03.126304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:03.126304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.126304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:03.126304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.126568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:03.126568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:03.126568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:03.126568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:03.126568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:03.126568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:03.126568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:03.126568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:03.126568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:03.126568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:03.126568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:03.126568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:03.126568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:03.126568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:03.126568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:03.126568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:03.126568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:03.126568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:03.126568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.126993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:03.127564 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:03.127564 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:03.127564 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:03.127564 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:03.127564 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:03.127564 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:03.127743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.127743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:03.127743 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:03.127743 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:03.127743 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:03.127743 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:03.127743 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:03.127743 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.127743 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:03.127743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.127743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:03.127743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:03.127743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:03.127743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:03.127743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:03.128087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:03.128087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:03.128087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:03.128087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:03.128087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:03.128087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:03.128087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:03.128087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:03.128087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:03.128087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:03.128087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:03.128087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:03.128087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:03.128087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:03.128428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:03.128428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:03.128428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:03.128428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:03.128428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:03.128428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.128428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:03.128428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.128428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:03.128428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:03.128428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:03.128428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:03.128428 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:03.128733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:03.129173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:03.129173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:03.129173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:03.129173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:03.129173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:03.129173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:03.129173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:03.129173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:03.129173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:03.129173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:03.129173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.129173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:03.850377 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:03.851033 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:03.851033 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:03.851721 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:03.889029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:03.889029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.889316 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:03.889316 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:03.889316 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:03.889316 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:03.889316 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:03.889316 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:03.889316 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.889316 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:03.890161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.890161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:03.890161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:03.890161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:03.890161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:03.890161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:03.890161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:03.890161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:03.890161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:03.890161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:03.890161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:03.890161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:03.890690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:03.890690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:03.890690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:03.890690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:03.890690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:03.890690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:03.890690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:03.890690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:03.890690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:03.890690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:03.890690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:03.890690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:03.890690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:03.891220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.891220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:03.891220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.891220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:03.891220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:03.891220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:03.891220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:03.891220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:03.891220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:03.891220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:03.891220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:03.891220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:03.891220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:03.891220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:03.891220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:03.891821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:03.891821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:03.891821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:03.891821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:03.891821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:03.891821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:03.891821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:03.891821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:03.891821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:03.891821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:03.891821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:03.891821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:03.891821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:03.891821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:03.891821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:03.891821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:03.891821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:03.891821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:03.891821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:03.892494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:03.892494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:03.892494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:03.892494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:03.892494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:03.892494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:07.648193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:07.648787 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:07.648787 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:07.650358 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:07.650358 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:07.650358 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:07.650358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.650631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:07.650631 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:07.650631 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:07.650631 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:07.650631 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:07.650631 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:07.650631 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.650631 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:07.651161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.651161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:07.651161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:07.651161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:07.651161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.651161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:07.651161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:07.651161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.651161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:07.651161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:07.651161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:07.651161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.651500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:07.651500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:07.651500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:07.651500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:07.651500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:07.651500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:07.651500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:07.651500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:07.651500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:07.651500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:07.651500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:07.651500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:07.651500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:07.651825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.651825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:07.651878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.651878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:07.651878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:07.651878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:07.651878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:07.651878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:07.651878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.651878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:07.651878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:07.651878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.651878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:07.651878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:07.651878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.651878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:07.651878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:07.651878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.652220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:07.652220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:07.652220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.652220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:07.652220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:07.652220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:07.652220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.652220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:07.652220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:07.652220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:07.652220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:07.652220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:07.652220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:07.652220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:07.652220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:07.652220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:07.652220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:07.652220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:07.652220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:07.652818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:07.652818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.652818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:07.653835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:07.653835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:07.653835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:07.654034 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:47:07.654802 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:07.654802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.654939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:07.654939 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:07.654939 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:07.654939 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:07.654939 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:07.654939 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:07.654939 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.654939 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:07.655384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.655384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:07.655384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:07.655384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:07.655384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.655384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:07.655384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:07.655384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:07.655384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.655384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:07.655384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:07.655384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:07.655384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:07.655384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:07.655384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:07.656074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:07.656074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:07.656074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:07.656074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:07.656074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:07.656074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:07.656074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:07.656074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.656074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:07.656074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.656074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:07.656074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:07.656074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:07.656074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.656750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:07.656750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:07.656750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:47:07.656750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:07.656750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:07.656750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.656750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:07.656750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:07.656750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:07.656750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.656750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:07.656750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:07.656750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:07.657398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:07.657398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:07.657398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:07.657398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:07.657398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:07.657398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:07.657398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:07.657398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:07.657398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:07.657398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:07.657398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.657398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:07.657942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.657942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:07.657942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:07.657942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:07.657942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:07.657942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:07.657942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.657942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:07.657942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:07.657942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.657942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:07.657942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:07.657942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.657942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:07.657942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:07.657942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.658427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:07.658427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:07.658427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:47:07.658427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:07.658427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:07.658427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.658427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:07.658427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:07.658427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:07.658427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.658427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:07.658427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:07.658427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:07.658427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:07.658427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:07.658427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:07.658427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:07.658427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:07.658829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:07.658829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:07.658829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:07.658829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:07.658829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:07.658829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.658829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:07.658829 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:07.659015 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:07.659131 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:07.659561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:07.659837 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:07.660147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.660689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:07.660689 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:07.660689 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:07.660689 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:07.660689 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:07.660689 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:07.660689 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.660689 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:07.661358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.661358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:07.661358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:07.661358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:07.661358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.661358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:07.661358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:07.661358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:07.661358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.661358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:07.661358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:07.661358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:07.661358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:07.661358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:07.661358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:07.662041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:07.662041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:07.662041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:07.662041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:07.662041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:07.662041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:07.662041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:07.662041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.662041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:07.662458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.662458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:07.662458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:07.662458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:07.662458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.662458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:07.662458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:07.662458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:07.662458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:07.662458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.662458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:07.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:07.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:07.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:07.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:07.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:07.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:07.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:07.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:07.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:07.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:07.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:07.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:07.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:07.663030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:07.663030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:07.663030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.663030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:07.663030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.663030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:07.663030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:07.663030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:07.663030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:07.663030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:07.663030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.663030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:07.663030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:07.663030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.663366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:07.663366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:07.663366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.663366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:07.663366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:07.663366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.663366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:07.663366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:07.663366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:07.663366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:07.663366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.663366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:07.663366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:07.663366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:07.663366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.663366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:07.663366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:07.663366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:07.663366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:07.663780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:07.663780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:07.663780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:07.663780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:07.663780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:07.663780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:07.663780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:07.663780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:07.663780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:07.663780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.663780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:07.668486 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:07.668734 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:07.668809 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:07.668956 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:07.669927 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:07.669927 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:07.669927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.670022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:07.670022 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:07.670022 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:07.670022 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:07.670022 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:07.670022 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:07.670022 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.670022 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:07.670522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.670522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:07.670522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:07.670522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:07.670522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.670522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:07.670522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:07.670522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.670522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:07.670522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:07.670522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:07.670522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.670839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:07.670839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:07.670839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:07.670839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:07.670839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:07.670839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:07.670839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:07.670839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:07.670839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:07.670839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:07.670839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:07.670839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:07.670839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:07.671120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.671120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:07.671454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.671454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:07.671454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:07.671454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:07.671454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:07.671454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:07.671454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.671454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:07.671454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:07.671454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.671454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:07.671454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:07.671454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.671454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:07.671454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:07.671454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:07.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:07.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:07.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:07.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:07.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:07.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:07.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:07.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:07.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:07.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:07.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:07.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:07.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:07.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:07.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:07.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:07.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:07.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:07.672771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:07.672771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:07.672771 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:08.850910 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:08.851539 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:08.851539 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:08.852069 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:08.879927 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:08.879927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:08.880130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:08.880130 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:08.880130 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:08.880130 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:08.880130 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:08.880130 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:08.880130 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:08.880130 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:08.880905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:08.880905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:08.880905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:08.880905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:08.880905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:08.880905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:08.880905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:08.880905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:08.880905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:08.880905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:08.880905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:08.880905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:08.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:08.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:08.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:08.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:08.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:08.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:08.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:08.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:08.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:08.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:08.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:08.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:08.881137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:08.881357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:08.881357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:08.881656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:08.881656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:08.881656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:08.881656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:08.881656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:08.881656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:08.881656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:08.881656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:08.881656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:08.881656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:08.881656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:08.881656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:08.881656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:08.881656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:08.881656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:08.881656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:08.881898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:08.881898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:08.881898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:08.881898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:08.881898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:08.881898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:08.881898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:08.881898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:08.881898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:08.881898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:08.881898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:08.881898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:08.881898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:08.881898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:08.881898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:08.881898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:08.881898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:08.881898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:08.881898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:08.882225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:08.882225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:08.882225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:12.648706 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:12.649284 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:12.649599 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:12.650873 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:12.650873 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:12.650873 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:12.650873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.651038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:12.651038 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:12.651038 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:12.651038 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:12.651038 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:12.651038 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:12.651038 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.651038 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:12.651655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.651655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:12.651655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:12.651655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:12.651655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.651655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:12.651655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:12.651655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.651655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:12.651655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:12.651655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:12.651655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.651933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:12.651933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:12.651933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:12.651933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:12.651933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:12.651933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:12.651933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:12.651933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:12.651933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:12.651933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:12.651933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:12.651933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:12.651933 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:12.652143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.652143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:12.652466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.652466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:12.652466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:12.652466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:12.652466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:12.652466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:12.652466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.652466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:12.652466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:12.652466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.652466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:12.652466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:12.652466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.652466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:12.652466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:12.652466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.653437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:12.653437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:12.653437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.653437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:12.653437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:12.653437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:12.653437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.653437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:12.653437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:12.653437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:12.653437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:12.653437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:12.653437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:12.653437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:12.653437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:12.653437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:12.653437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:12.653437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:12.653437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:12.653883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:12.653883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.653883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:12.653883 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:12.654119 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:12.654119 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:12.654289 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:47:12.654550 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:12.655034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.655146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:12.655146 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:12.655146 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:12.655146 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:12.655146 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:12.655146 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:12.655146 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.655146 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:12.655687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.655687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:12.655687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:12.655687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:12.655687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.655687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:12.655687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:12.655687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:12.655687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.655687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:12.655687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:12.655687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:12.655687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:12.655687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:12.655687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:12.655924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:12.655924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:12.655924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:12.655924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:12.655924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:12.655924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:12.655924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:12.655924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.655924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:12.656246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.656246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:12.656246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:12.656246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:12.656246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.656246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:12.656246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:12.656246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:47:12.656246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:12.656246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:12.656246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.656246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:12.656469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:12.656469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:12.656469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.656469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:12.656469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:12.656469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:12.656469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:12.656469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:12.656469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:12.656469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:12.656469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:12.656469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:12.656469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:12.656469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:12.656673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:12.656673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:12.656673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.656673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:12.656983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.656983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:12.656983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:12.656983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:12.656983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:12.656983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:12.656983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.656983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:12.656983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:12.656983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.656983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:12.656983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:12.656983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.656983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:12.656983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:12.656983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.657750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:12.657750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:12.657750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:47:12.657750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:12.657750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:12.657750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.657750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:12.657750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:12.657750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:12.657750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.657750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:12.657750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:12.657750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:12.657750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:12.657750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:12.657750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:12.657750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:12.657750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:12.658141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:12.658141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:12.658141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:12.658141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:12.658141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:12.658141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.658141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:12.659037 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:12.659230 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:12.659346 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:12.659480 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:12.660158 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:12.660158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.660319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:12.660319 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:12.660319 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:12.660319 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:12.660319 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:12.660319 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:12.660319 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.660319 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:12.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:12.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:12.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:12.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:12.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:12.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:12.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:12.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:12.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:12.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:12.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:12.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:12.661224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:12.661224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:12.661224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:12.661224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:12.661224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:12.661224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:12.661224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:12.661224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.661224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:12.661798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.661798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:12.661798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:12.661798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:12.661798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.661798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:12.661798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:12.661798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:12.661798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:12.661798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.661798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:12.662017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:12.662017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:12.662017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.662017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:12.662017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:12.662017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:12.662017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:12.662017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:12.662017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:12.662017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:12.662017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:12.662017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:12.662017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:12.662017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:12.662288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:12.662288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:12.662288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.662288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:12.662780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.662780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:12.662780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:12.662780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:12.662780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:12.662780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:12.662780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.662780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:12.662780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:12.662780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.662780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:12.662780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:12.662780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.662780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:12.662780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:12.662780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.662780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:12.663109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:12.663109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:12.663109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:12.663109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.663109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:12.663109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:12.663109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:12.663109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.663109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:12.663109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:12.663109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:12.663109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:12.663109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:12.663109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:12.663109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:12.663109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:12.663109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:12.663109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:12.663109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:12.663483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:12.663483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:12.663483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.663483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:12.668868 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:12.669126 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:12.669216 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:12.669408 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:12.670401 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:12.670401 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:12.670401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.670536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:12.670536 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:12.670536 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:12.670536 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:12.670536 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:12.670536 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:12.670536 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.670536 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:12.671054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.671054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:12.671054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:12.671054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:12.671054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.671054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:12.671054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:12.671054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.671054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:12.671054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:12.671054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:12.671054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.671445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:12.671445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:12.671445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:12.671445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:12.671445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:12.671445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:12.671445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:12.671445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:12.671445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:12.671445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:12.671445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:12.671445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:12.671445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:12.671696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.671696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:12.671976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.671976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:12.671976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:12.671976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:12.671976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:12.671976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:12.671976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.671976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:12.671976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:12.671976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.671976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:12.671976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:12.671976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.671976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:12.671976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:12.671976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:12.672955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:12.672955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:12.672955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.672955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:12.672955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:12.672955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:12.672955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:12.672955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:12.672955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:12.672955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:12.672955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:12.672955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:12.672955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:12.672955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:12.672955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:12.672955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:12.672955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:12.672955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:12.672955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:12.673221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:12.673221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:12.673221 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:13.851387 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:13.851954 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:13.851954 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:13.852373 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:13.876694 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:13.876694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:13.876853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:13.876853 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:13.876853 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:13.876853 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:13.876853 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:13.876853 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:13.876853 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:13.876853 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:13.877455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:13.877455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:13.877455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:13.877455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:13.877455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:13.877455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:13.877455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:13.877455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:13.877455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:13.877455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:13.877455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:13.877455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:13.877705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:13.877705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:13.877705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:13.877705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:13.877705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:13.877705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:13.877705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:13.877705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:13.877705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:13.877705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:13.877705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:13.877705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:13.877705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:13.877985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:13.877985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:13.878267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:13.878267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:13.878267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:13.878267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:13.878267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:13.878267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:13.878267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:13.878267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:13.878267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:13.878267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:13.878267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:13.878267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:13.878267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:13.878267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:13.878267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:13.878267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:13.879056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:13.879056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:13.879056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:13.879056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:13.879056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:13.879056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:13.879056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:13.879056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:13.879056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:13.879056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:13.879056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:13.879056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:13.879056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:13.879056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:13.879056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:13.879056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:13.879056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:13.879056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:13.879056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:13.879552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:13.879552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:13.879552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:17.649245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:17.649752 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:17.649752 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:17.650354 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:17.651696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:17.651696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:17.651696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.651696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:17.651696 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:17.651696 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:17.651696 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:17.651696 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:17.651696 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:17.651696 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.651696 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:17.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:17.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:17.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:17.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:17.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:17.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:17.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:17.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:17.652654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.652955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:17.652955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:17.652955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:17.652955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:17.652955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:17.652955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:17.652955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:17.652955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:17.652955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:17.652955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:17.652955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:17.652955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:17.652955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:17.653230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.653230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:17.653643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.653643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:17.653643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:17.653643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:17.653643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:17.653643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:17.653643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.653643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:17.653643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:17.653643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.653643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:17.653643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:17.653643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.653643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:17.653643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:17.653643 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.655036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:17.655036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:17.655036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.655036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:17.655036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:17.655036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:17.655036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.655036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:17.655036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:17.655036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:17.655036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:17.655036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:17.655036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:17.655036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:17.655036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:17.655036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:17.655036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:17.655036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:17.655036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:17.655663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:17.655663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.655663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:17.655663 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:17.655663 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:17.655663 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:17.655663 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:47:17.655663 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:17.655901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.655901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:17.655901 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:17.655901 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:17.655901 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:17.655901 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:17.655901 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:17.655901 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.655901 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:17.656584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.656584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:17.656584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:17.656584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:17.656584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.656584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:17.656584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:17.656584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:17.656584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.656584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:17.656584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:17.656584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:17.656584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:17.656584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:17.656584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:17.656975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:17.656975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:17.656975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:17.656975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:17.656975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:17.656975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:17.656975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:17.656975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.656975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:17.656975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.656975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:17.656975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:17.656975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:17.656975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.657308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:17.657308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:17.657308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:47:17.657308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:17.657308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:17.657308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.657308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:17.657308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:17.657308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:17.657308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.657308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:17.657308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:17.657308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:17.658156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:17.658156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:17.658156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:17.658156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:17.658156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:17.658156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:17.658156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:17.658156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:17.658156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:17.658156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:17.658156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.658156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:17.658778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.658778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:17.658778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:17.658778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:17.658778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:17.658778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:17.658778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.658778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:17.658778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:17.658778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.658778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:17.658778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:17.658778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.658778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:17.658778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:17.658778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.659224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:17.659224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:17.659224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:47:17.659224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:17.659224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:17.659224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.659224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:17.659224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:17.659224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:17.659224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.659224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:17.659224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:17.659224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:17.659224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:17.659224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:17.659224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:17.659224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:17.659224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:17.659687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:17.659687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:17.659687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:17.659687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:17.659687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:17.659687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.659687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:17.659687 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:17.660023 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:17.660100 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:17.660460 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:17.660743 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:17.660999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.661499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:17.661499 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:17.661499 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:17.661499 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:17.661499 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:17.661499 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:17.661499 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.661499 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:17.662350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.662350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:17.662350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:17.662350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:17.662350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.662350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:17.662350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:17.662350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:17.662350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.662350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:17.662350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:17.662350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:17.662350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:17.662350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:17.662350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:17.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:17.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:17.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:17.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:17.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:17.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:17.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:17.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.662714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:17.663306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.663306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:17.663306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:17.663306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:17.663306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.663306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:17.663306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:17.663306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:17.663306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:17.663306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.663306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:17.663800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:17.663800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:17.663800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.663800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:17.663800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:17.663800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:17.663800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:17.663800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:17.663800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:17.663800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:17.663800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:17.663800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:17.663800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:17.663800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:17.664120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:17.664120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:17.664120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.664120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:17.664431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.664431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:17.664431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:17.664431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:17.664431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:17.664431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:17.664431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.664431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:17.664431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:17.664431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.664431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:17.664431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:17.664431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.664431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:17.664431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:17.664431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.664431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:17.664888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:17.664888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:17.664888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:17.664888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.664888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:17.664888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:17.664888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:17.664888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.664888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:17.664888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:17.664888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:17.664888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:17.664888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:17.664888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:17.664888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:17.664888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:17.664888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:17.664888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:17.664888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:17.665590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:17.665590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:17.665590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.665590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:17.669288 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:17.669631 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:17.669768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:17.669942 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:17.670401 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:17.670639 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:17.670793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.671324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:17.671324 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:17.671324 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:17.671324 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:17.671324 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:17.671324 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:17.671324 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.671324 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:17.672177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.672177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:17.672177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:17.672177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:17.672177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.672177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:17.672177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:17.672177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.672177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:17.672177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:17.672177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:17.672177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.672563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:17.672563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:17.672563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:17.672563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:17.672563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:17.672563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:17.672563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:17.672563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:17.672563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:17.672563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:17.672563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:17.672563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:17.672563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:17.672970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.672970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:17.673282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.673282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:17.673282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:17.673282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:17.673282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:17.673282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:17.673282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.673282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:17.673282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:17.673282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.673282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:17.673282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:17.673282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.673282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:17.673282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:17.673282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:17.673636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:17.673636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:17.673636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.673636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:17.673636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:17.673636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:17.673636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:17.673636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:17.673636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:17.673636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:17.673636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:17.673636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:17.673636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:17.673636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:17.673636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:17.673636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:17.673636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:17.673636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:17.673636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:17.673944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:17.673944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:17.673944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:18.851793 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:18.852270 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:18.852270 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:18.852940 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:18.884920 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:18.885129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:18.885686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:18.885686 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:18.885686 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:18.885686 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:18.885686 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:18.885686 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:18.885686 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:18.885686 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:18.886580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:18.886580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:18.886580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:18.886580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:18.886580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:18.886580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:18.886580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:18.886580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:18.886580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:18.886580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:18.886580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:18.886580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:18.886823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:18.886823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:18.886823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:18.886823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:18.886823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:18.886823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:18.886823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:18.886823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:18.886823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:18.886823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:18.886823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:18.886823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:18.886823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:18.887020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:18.887020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:18.887641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:18.887641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:18.887641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:18.887641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:18.887641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:18.887641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:18.887641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:18.887641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:18.887641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:18.887641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:18.887641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:18.887641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:18.887641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:18.887641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:18.887641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:18.887641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:18.888107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:18.888107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:18.888107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:18.888107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:18.888107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:18.888107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:18.888107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:18.888107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:18.888107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:18.888107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:18.888107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:18.888107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:18.888107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:18.888107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:18.888107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:18.888107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:18.888107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:18.888107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:18.888107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:18.888606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:18.888606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:18.888606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:22.649795 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:22.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:22.650385 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:22.650929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:22.651348 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:22.651545 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:22.651725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.652450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:22.652450 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:22.652450 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:22.652450 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:22.652450 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:22.652450 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:22.652450 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.652450 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:22.653267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.653267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:22.653267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:22.653267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:22.653267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.653267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:22.653267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:22.653267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.653267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:22.653267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:22.653267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:22.653267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.653667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:22.653667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:22.653667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:22.653667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:22.653667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:22.653667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:22.653667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:22.653667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:22.653667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:22.653667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:22.653667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:22.653667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:22.653667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:22.654049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.654049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:22.654368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.654368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:22.654368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:22.654368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:22.654368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:22.654368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:22.654368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.654368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:22.654368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:22.654368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.654368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:22.654368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:22.654368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.654368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:22.654368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:22.654368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.654708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:22.654708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:22.654708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.654708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:22.654708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:22.654708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:22.654708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.654708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:22.654708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:22.654708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:22.654708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:22.654708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:22.654708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:22.654708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:22.654708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:22.654708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:22.654708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:22.654708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:22.654708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:22.655027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:22.655027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.655027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:22.655298 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:22.655475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:22.655585 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:22.655729 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:47:22.656330 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:22.656606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.657224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:22.657224 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:22.657224 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:22.657224 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:22.657224 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:22.657224 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:22.657224 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.657224 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:22.657930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.657930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:22.657930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:22.657930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:22.657930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.657930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:22.657930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:22.657930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:22.657930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.657930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:22.657930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:22.657930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:22.657930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:22.657930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:22.657930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:22.658390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:22.658390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:22.658390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:22.658390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:22.658390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:22.658390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:22.658390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:22.658390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.658390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:22.658826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.658826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:22.658826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:22.658826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:22.658826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.658826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:22.658826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:22.658826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:47:22.658826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:22.658826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:22.658826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.658826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:22.659157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:22.659157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:22.659157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.659157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:22.659157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:22.659157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:22.659157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:22.659157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:22.659157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:22.659157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:22.659157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:22.659157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:22.659157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:22.659157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:22.659520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:22.659520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:22.659520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.659520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:22.659991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.659991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:22.659991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:22.659991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:22.659991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:22.659991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:22.659991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.659991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:22.659991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:22.659991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.659991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:22.659991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:22.659991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.659991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:22.659991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:22.659991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.660296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:22.660296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:22.660296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:47:22.660296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:22.660296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:22.660296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.660296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:22.660296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:22.660296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:22.660296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.660296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:22.660296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:22.660296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:22.660296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:22.660296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:22.660296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:22.660296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:22.660296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:22.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:22.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:22.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:22.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:22.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:22.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.660682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:22.660994 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:22.661157 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:22.661287 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:22.661425 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:22.662030 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:22.662364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.662966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:22.662966 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:22.662966 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:22.662966 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:22.662966 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:22.662966 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:22.662966 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.662966 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:22.663666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.663666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:22.663666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:22.663666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:22.663666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.663666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:22.663666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:22.663666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:22.663666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.663666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:22.663666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:22.663666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:22.663666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:22.663666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:22.663666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:22.664044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:22.664044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:22.664044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:22.664044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:22.664044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:22.664044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:22.664044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:22.664044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.664044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:22.664573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.664573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:22.664573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:22.664573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:22.664573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.664573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:22.664573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:22.664573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:22.664573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:22.664573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.664573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:22.664909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:22.664909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:22.664909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.664909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:22.664909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:22.664909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:22.664909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:22.664909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:22.664909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:22.664909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:22.664909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:22.664909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:22.664909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:22.664909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:22.665273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:22.665273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:22.665273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.665273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:22.665696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.665696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:22.665696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:22.665696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:22.665696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:22.665696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:22.665696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.665696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:22.665696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:22.665696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.665696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:22.665696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:22.665696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.665696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:22.665696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:22.665696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.665696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:22.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:22.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:22.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:22.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:22.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:22.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:22.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:22.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:22.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:22.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:22.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:22.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:22.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:22.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:22.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:22.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:22.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:22.666618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:22.666618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:22.666618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.666618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:22.669722 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:22.669890 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:22.670002 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:22.670151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:22.670723 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:22.671029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:22.671229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:22.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:22.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:22.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:22.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:22.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:22.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.671885 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:22.672917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.672917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:22.672917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:22.672917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:22.672917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.672917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:22.672917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:22.672917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.672917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:22.672917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:22.672917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:22.672917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.673197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:22.673197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:22.673197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:22.673197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:22.673197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:22.673197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:22.673197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:22.673197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:22.673197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:22.673197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:22.673197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:22.673197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:22.673197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:22.673504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.673504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:22.674021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.674021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:22.674021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:22.674021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:22.674021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:22.674021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:22.674021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.674021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:22.674021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:22.674021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.674021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:22.674021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:22.674021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.674021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:22.674021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:22.674021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:22.674418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:22.674418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:22.674418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.674418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:22.674418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:22.674418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:22.674418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:22.674418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:22.674418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:22.674418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:22.674418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:22.674418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:22.674418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:22.674418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:22.674418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:22.674418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:22.674418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:22.674418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:22.674418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:22.674897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:22.674897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:22.674897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:23.852242 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:23.852768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:23.852768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:23.853358 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:23.883509 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:23.883724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:23.884247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:23.884247 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:23.884247 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:23.884247 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:23.884247 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:23.884247 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:23.884247 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:23.884247 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:23.885003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:23.885003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:23.885003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:23.885003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:23.885003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:23.885003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:23.885003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:23.885003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:23.885003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:23.885003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:23.885003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:23.885003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:23.885265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:23.885265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:23.885265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:23.885265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:23.885265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:23.885265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:23.885265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:23.885265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:23.885265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:23.885265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:23.885265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:23.885265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:23.885265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:23.885506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:23.885506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:23.885762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:23.885762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:23.885762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:23.885762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:23.885762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:23.885762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:23.885762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:23.885762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:23.885762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:23.885762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:23.885762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:23.885762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:23.885762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:23.885762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:23.885762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:23.885762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:23.885995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:23.885995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:23.885995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:23.885995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:23.885995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:23.885995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:23.885995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:23.885995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:23.885995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:23.885995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:23.885995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:23.885995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:23.885995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:23.885995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:23.885995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:23.885995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:23.885995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:23.885995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:23.885995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:23.886310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:23.886310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:23.886310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:27.650023 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:27.650643 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:27.650643 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:27.651386 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:27.651514 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:27.651853 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:27.651967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.652540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:27.652540 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:27.652540 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:27.652540 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:27.652540 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:27.652540 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:27.652540 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.652540 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:27.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:27.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:27.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:27.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:27.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:27.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:27.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:27.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:27.653140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.653567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:27.653567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:27.653567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:27.653567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:27.653567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:27.653567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:27.653567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:27.653567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:27.653567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:27.653567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:27.653567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:27.653567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:27.653567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:27.654025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.654025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:27.654025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.654025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:27.654025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:27.654025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:27.654025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:27.654025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:27.654025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.654025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:27.654025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:27.654025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.654025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:27.654025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:27.654025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.654509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:27.654509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:27.654509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.654509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:27.654509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:27.654509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.654509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:27.654509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:27.654509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:27.654509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.654509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:27.654509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:27.654509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:27.654509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:27.654509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:27.654509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:27.654509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:27.654509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:27.654509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:27.655118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:27.655118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:27.655118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:27.655118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:27.655118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.655118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:27.655495 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:27.655718 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:27.655811 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:27.655923 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:47:27.656201 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:27.656432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:27.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:27.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:27.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:27.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:27.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:27.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:27.657654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.657654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:27.657654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:27.657654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:27.657654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.657654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:27.657654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:27.657654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:27.657654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.657654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:27.657654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:27.657654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:27.657654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:27.657654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:27.657654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:27.658116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:27.658116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:27.658116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:27.658116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:27.658116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:27.658116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:27.658116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:27.658116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.658116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:27.658497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.658497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:27.658497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:27.658497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:27.658497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.658497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:27.658497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:27.658497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:47:27.658497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:27.658497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:27.658497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.658497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:27.658883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:27.658883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:27.658883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.658883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:27.658883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:27.658883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:27.658883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:27.658883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:27.658883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:27.658883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:27.658883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:27.658883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:27.658883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:27.658883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:27.659184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:27.659184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:27.659184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.659184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:27.659336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.659336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:27.659336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:27.659336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:27.659336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:27.659336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:27.659336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.659336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:27.659336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:27.659336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.659336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:27.659336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:27.659336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.659336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:27.659336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:27.659336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.659912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:27.659912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:27.659912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:47:27.659912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:27.659912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:27.659912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.659912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:27.659912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:27.659912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:27.659912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.659912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:27.659912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:27.659912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:27.659912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:27.659912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:27.659912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:27.659912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:27.659912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:27.660208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:27.660208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:27.660208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:27.660208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:27.660208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:27.660208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.660208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:27.661188 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:27.661392 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:27.661518 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:27.661641 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:27.661959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:27.662227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.662814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:27.662814 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:27.662814 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:27.662814 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:27.662814 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:27.662814 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:27.662814 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.662814 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:27.663400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.663400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:27.663400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:27.663400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:27.663400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.663400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:27.663400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:27.663400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:27.663400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.663400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:27.663400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:27.663400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:27.663400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:27.663400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:27.663400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:27.663675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:27.663675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:27.663675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:27.663675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:27.663675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:27.663675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:27.663675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:27.663675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.663675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:27.663855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.663855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:27.663855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:27.663855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:27.663855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.663855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:27.663855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:27.663855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:27.663855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:27.663855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.663855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:27.664087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:27.664087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:27.664087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.664087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:27.664087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:27.664087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:27.664087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:27.664087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:27.664087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:27.664087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:27.664087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:27.664087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:27.664087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:27.664087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:27.664344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:27.664344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:27.664344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.664344 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:27.664444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.664444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:27.664444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:27.664444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:27.664444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:27.664444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:27.664444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.664444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:27.664444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:27.664444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.664444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:27.664444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:27.664444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.664444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:27.664444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:27.664444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.664444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:27.664787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:27.664787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:27.664787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:27.664787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.664787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:27.664787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:27.664787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:27.664787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.664787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:27.664787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:27.664787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:27.664787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:27.664787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:27.664787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:27.664787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:27.664787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:27.664787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:27.664787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:27.664787 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:27.665123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:27.665123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:27.665123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.665123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:27.669959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:27.670163 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:27.670297 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:27.670457 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:27.670924 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:27.671199 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:27.671395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.672110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:27.672110 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:27.672110 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:27.672110 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:27.672110 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:27.672110 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:27.672110 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.672110 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:27.672989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.672989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:27.672989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:27.672989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:27.672989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.672989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:27.672989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:27.672989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.672989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:27.672989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:27.672989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:27.672989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.673227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:27.673227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:27.673227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:27.673227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:27.673227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:27.673227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:27.673227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:27.673227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:27.673227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:27.673227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:27.673227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:27.673227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:27.673227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:27.673455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.673455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:27.673978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.673978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:27.673978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:27.673978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:27.673978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:27.673978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:27.673978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.673978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:27.673978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:27.673978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.673978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:27.673978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:27.673978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.673978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:27.673978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:27.673978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:27.674324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:27.674324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:27.674324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.674324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:27.674324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:27.674324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:27.674324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:27.674324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:27.674324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:27.674324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:27.674324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:27.674324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:27.674324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:27.674324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:27.674324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:27.674324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:27.674324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:27.674324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:27.674324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:27.674616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:27.674616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:27.674616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:28.852696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:28.853183 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:28.853183 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:28.853940 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:28.892224 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:28.892540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:28.893276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:28.893276 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:28.893276 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:28.893276 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:28.893276 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:28.893276 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:28.893276 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:28.893276 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:28.894274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:28.894274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:28.894274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:28.894274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:28.894274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:28.894274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:28.894274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:28.894274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:28.894274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:28.894274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:28.894274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:28.894274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:28.894546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:28.894546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:28.894546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:28.894546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:28.894546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:28.894546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:28.894546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:28.894546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:28.894546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:28.894546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:28.894546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:28.894546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:28.894546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:28.894778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:28.894778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:28.895067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:28.895067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:28.895067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:28.895067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:28.895067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:28.895067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:28.895067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:28.895067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:28.895067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:28.895067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:28.895067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:28.895067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:28.895067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:28.895067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:28.895067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:28.895067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:28.895756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:28.895756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:28.895756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:28.895756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:28.895756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:28.895756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:28.895756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:28.895756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:28.895756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:28.895756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:28.895756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:28.895756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:28.895756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:28.895756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:28.895756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:28.895756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:28.895756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:28.895756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:28.895756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:28.896411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:28.896411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:28.896411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:32.650613 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:32.651168 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:32.651168 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:32.651570 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:32.651745 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:32.651859 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:32.651998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.652458 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:32.653111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.653111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:32.653111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:32.653111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:32.653111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.653111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:32.653111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:32.653111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.653111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:32.653111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:32.653111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:32.653111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.653461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:32.653461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:32.653461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:32.653461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:32.653461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:32.653461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:32.653461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:32.653461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:32.653461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:32.653461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:32.653461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:32.653461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:32.653461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:32.653829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.653829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:32.654046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.654046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:32.654046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:32.654046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:32.654046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:32.654046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:32.654046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.654046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:32.654046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:32.654046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.654046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:32.654046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:32.654046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.654046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:32.654046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:32.654046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.654484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:32.654484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:32.654484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.654484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:32.654484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:32.654484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:32.654484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.654484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:32.654484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:32.654484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:32.654484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:32.654484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:32.654484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:32.654484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:32.654484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:32.654484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:32.654484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:32.654484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:32.654484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:32.654960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:32.654960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.654960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:32.655793 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:32.656035 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:32.656164 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:32.656335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:47:32.656825 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:32.657031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.657530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:32.657530 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:32.657530 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:32.657530 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:32.657530 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:32.657530 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:32.657530 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.657530 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:32.658398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.658398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:32.658398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:32.658398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:32.658398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.658398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:32.658398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:32.658398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:32.658398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.658398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:32.658398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:32.658398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:32.658398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:32.658398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:32.658398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:32.658794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:32.658794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:32.658794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:32.658794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:32.658794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:32.658794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:32.658794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:32.658794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.658794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:32.659350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.659350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:32.659350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:32.659350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:32.659350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.659350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:32.659350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:32.659350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:47:32.659350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:32.659350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:32.659350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.659350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:32.659712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:32.659712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:32.659712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.659712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:32.659712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:32.659712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:32.659712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:32.659712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:32.659712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:32.659712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:32.659712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:32.659712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:32.659712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:32.659712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:32.660104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:32.660104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:32.660104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.660104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:32.660462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.660462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:32.660462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:32.660462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:32.660462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:32.660462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:32.660462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.660462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:32.660462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:32.660462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.660462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:32.660462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:32.660462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.660462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:32.660462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:32.660462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.660899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:32.660899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:32.660899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:47:32.660899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:32.660899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:32.660899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.660899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:32.660899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:32.660899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:32.660899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.660899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:32.660899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:32.660899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:32.660899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:32.660899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:32.660899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:32.660899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:32.660899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:32.661362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:32.661362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:32.661362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:32.661362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:32.661362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:32.661362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.661362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:32.661546 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:32.661618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:32.661734 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:32.661888 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:32.662377 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:32.662569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.663052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:32.663052 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:32.663052 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:32.663052 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:32.663052 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:32.663052 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:32.663052 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.663052 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:32.663779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.663779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:32.663779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:32.663779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:32.663779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.663779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:32.663779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:32.663779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:32.663779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.663779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:32.663779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:32.663779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:32.663779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:32.663779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:32.663779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:32.664154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:32.664154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:32.664154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:32.664154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:32.664154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:32.664154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:32.664154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:32.664154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.664154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:32.664595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.664595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:32.664595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:32.664595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:32.664595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.664595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:32.664595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:32.664595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:32.664595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:32.664595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.664595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:32.664887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:32.664887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:32.664887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.664887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:32.664887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:32.664887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:32.664887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:32.664887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:32.664887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:32.664887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:32.664887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:32.664887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:32.664887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:32.664887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:32.665274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:32.665274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:32.665274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.665274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:32.665586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.665586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:32.665586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:32.665586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:32.665586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:32.665586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:32.665586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.665586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:32.665586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:32.665586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.665586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:32.665586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:32.665586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.665586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:32.665586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:32.665586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.665586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:32.666033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:32.666033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:32.666033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:32.666033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.666033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:32.666033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:32.666033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:32.666033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.666033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:32.666033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:32.666033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:32.666033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:32.666033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:32.666033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:32.666033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:32.666033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:32.666033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:32.666033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:32.666033 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:32.666555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:32.666555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:32.666555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.666555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:32.670268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:32.670530 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:32.670644 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:32.670782 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:32.671213 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:32.671496 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:32.671663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.672182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:32.672182 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:32.672182 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:32.672182 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:32.672182 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:32.672182 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:32.672182 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.672182 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:32.672862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.672862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:32.672862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:32.672862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:32.672862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.672862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:32.672862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:32.672862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.672862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:32.672862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:32.672862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:32.672862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.673194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:32.673194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:32.673194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:32.673194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:32.673194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:32.673194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:32.673194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:32.673194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:32.673194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:32.673194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:32.673194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:32.673194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:32.673194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:32.673523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.673523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:32.673826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.673826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:32.673826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:32.673826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:32.673826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:32.673826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:32.673826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.673826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:32.673826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:32.673826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.673826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:32.673826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:32.673826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.673826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:32.673826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:32.673826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:32.674270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:32.674270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:32.674270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.674270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:32.674270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:32.674270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:32.674270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:32.674270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:32.674270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:32.674270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:32.674270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:32.674270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:32.674270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:32.674270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:32.674270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:32.674270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:32.674270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:32.674270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:32.674270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:32.674750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:32.674750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:32.674750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:33.853269 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:33.853917 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:33.853917 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:33.854492 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:33.890968 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:33.891225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:33.891874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:33.891874 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:33.891874 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:33.891874 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:33.891874 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:33.891874 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:33.891874 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:33.891874 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:33.892589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:33.892589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:33.892589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:33.892589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:33.892589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:33.892589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:33.892589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:33.892589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:33.892589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:33.892589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:33.892589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:33.892589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:33.892952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:33.892952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:33.892952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:33.892952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:33.892952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:33.892952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:33.892952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:33.892952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:33.892952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:33.892952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:33.892952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:33.892952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:33.892952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:33.893321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:33.893321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:33.893741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:33.893741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:33.893741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:33.893741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:33.893741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:33.893741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:33.893741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:33.893741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:33.893741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:33.893741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:33.893741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:33.893741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:33.893741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:33.893741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:33.893741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:33.893741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:33.894166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:33.894166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:33.894166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:33.894166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:33.894166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:33.894166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:33.894166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:33.894166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:33.894166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:33.894166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:33.894166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:33.894166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:33.894166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:33.894166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:33.894166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:33.894166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:33.894166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:33.894166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:33.894166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:33.894724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:33.894724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:33.894724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:37.651012 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:37.651465 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:37.651465 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:37.652016 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:37.652228 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:37.652399 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:37.652517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.653008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:37.653008 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:37.653008 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:37.653008 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:37.653008 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:37.653008 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:37.653008 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.653008 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:37.653900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.653900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:37.653900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:37.653900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:37.653900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.653900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:37.653900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:37.653900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.653900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:37.653900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:37.653900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:37.653900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.654308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:37.654308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:37.654308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:37.654308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:37.654308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:37.654308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:37.654308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:37.654308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:37.654308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:37.654308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:37.654308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:37.654308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:37.654308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:37.655043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.655043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:37.655136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.655136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:37.655136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:37.655136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:37.655136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:37.655136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:37.655136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.655136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:37.655136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:37.655136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.655136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:37.655136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:37.655136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.655136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:37.655136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:37.655136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.655555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:37.655555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:37.655555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.655555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:37.655555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:37.655555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:37.655555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.655555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:37.655555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:37.655555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:37.655555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:37.655555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:37.655555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:37.655555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:37.655555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:37.655555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:37.655555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:37.655555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:37.655555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:37.655979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:37.655979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.655979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:37.656274 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:37.656450 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:37.656608 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:37.656739 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:47:37.657552 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:37.657756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.658217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:37.658217 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:37.658217 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:37.658217 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:37.658217 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:37.658217 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:37.658217 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.658217 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:37.658960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.658960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:37.658960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:37.658960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:37.658960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.658960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:37.658960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:37.658960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:37.658960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.658960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:37.658960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:37.658960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:37.658960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:37.658960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:37.658960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:37.659436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:37.659436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:37.659436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:37.659436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:37.659436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:37.659436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:37.659436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:37.659436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.659436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:37.659872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.659872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:37.659872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:37.659872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:37.659872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.659872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:37.659872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:37.659872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:47:37.659872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:37.659872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:37.659872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.659872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:37.660219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:37.660219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:37.660219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.660219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:37.660219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:37.660219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:37.660219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:37.660219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:37.660219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:37.660219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:37.660219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:37.660219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:37.660219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:37.660219 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:37.660606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:37.660606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:37.660606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.660606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:37.661003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.661003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:37.661003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:37.661003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:37.661003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:37.661003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:37.661003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.661003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:37.661003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:37.661003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.661003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:37.661003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:37.661003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.661003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:37.661003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:37.661003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.661386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:37.661386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:37.661386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:47:37.661386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:37.661386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:37.661386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.661386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:37.661386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:37.661386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:37.661386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.661386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:37.661386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:37.661386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:37.661386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:37.661386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:37.661386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:37.661386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:37.661386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:37.662151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:37.662151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:37.662151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:37.662151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:37.662151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:37.662151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.662151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:37.662151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:37.662151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:37.662151 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:37.662421 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:37.662680 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:37.662906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.663326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:37.663326 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:37.663326 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:37.663326 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:37.663326 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:37.663326 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:37.663326 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.663326 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:37.663904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.663904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:37.663904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:37.663904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:37.663904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.663904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:37.663904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:37.663904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:37.663904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.663904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:37.663904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:37.663904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:37.663904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:37.663904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:37.663904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:37.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:37.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:37.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:37.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:37.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:37.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:37.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:37.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:37.664716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.664716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:37.664716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:37.664716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:37.664716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.664716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:37.664716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:37.664716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:37.664716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:37.664716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.664716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:37.665061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:37.665061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:37.665061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.665061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:37.665061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:37.665061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:37.665061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:37.665061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:37.665061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:37.665061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:37.665061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:37.665061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:37.665061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:37.665061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:37.665408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:37.665408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:37.665408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.665408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:37.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:37.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:37.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:37.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:37.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:37.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:37.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:37.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:37.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:37.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:37.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:37.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:37.666326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:37.666326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:37.666326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:37.666326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.666326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:37.666326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:37.666326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:37.666326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.666326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:37.666326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:37.666326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:37.666326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:37.666326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:37.666326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:37.666326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:37.666326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:37.666326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:37.666326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:37.666326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:37.666627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:37.666627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:37.666627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.666627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:37.670709 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:37.670912 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:37.671053 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:37.671234 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:37.671900 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:37.672245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:37.672507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.673268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:37.673268 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:37.673268 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:37.673268 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:37.673268 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:37.673268 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:37.673268 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.673268 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:37.674100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.674100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:37.674100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:37.674100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:37.674100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.674100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:37.674100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:37.674100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.674100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:37.674100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:37.674100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:37.674100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.674502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:37.674502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:37.674502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:37.674502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:37.674502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:37.674502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:37.674502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:37.674502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:37.674502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:37.674502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:37.674502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:37.674502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:37.674502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:37.674831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.674831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:37.675292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.675292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:37.675292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:37.675292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:37.675292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:37.675292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:37.675292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.675292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:37.675292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:37.675292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.675292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:37.675292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:37.675292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.675292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:37.675292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:37.675292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:37.675806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:37.675806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:37.675806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.675806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:37.675806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:37.675806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:37.675806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:37.675806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:37.675806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:37.675806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:37.675806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:37.675806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:37.675806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:37.675806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:37.675806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:37.675806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:37.675806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:37.675806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:37.675806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:37.676184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:37.676184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:37.676184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:38.853721 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:38.854117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:38.854117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:38.854613 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:38.884273 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:38.884464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:38.884926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:38.884926 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:38.884926 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:38.884926 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:38.884926 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:38.884926 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:38.884926 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:38.884926 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:38.885575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:38.885575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:38.885575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:38.885575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:38.885575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:38.885575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:38.885575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:38.885575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:38.885575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:38.885575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:38.885575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:38.885575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:38.885850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:38.885850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:38.885850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:38.885850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:38.885850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:38.885850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:38.885850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:38.885850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:38.885850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:38.885850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:38.885850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:38.885850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:38.885850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:38.886100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:38.886100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:38.886282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:38.886282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:38.886282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:38.886282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:38.886282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:38.886282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:38.886282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:38.886282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:38.886282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:38.886282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:38.886282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:38.886282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:38.886282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:38.886282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:38.886282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:38.886282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:38.886589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:38.886589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:38.886589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:38.886589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:38.886589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:38.886589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:38.886589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:38.886589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:38.886589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:38.886589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:38.886589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:38.886589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:38.886589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:38.886589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:38.886589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:38.886589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:38.886589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:38.886589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:38.886589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:38.886895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:38.886895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:38.886895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:42.651450 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:42.651872 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:42.651872 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:42.652441 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:42.652583 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:42.652702 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:42.652799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.653269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:42.653269 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:42.653269 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:42.653269 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:42.653269 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:42.653269 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:42.653269 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.653269 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:42.653920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.653920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:42.653920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:42.653920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:42.653920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.653920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:42.653920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:42.653920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.653920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:42.653920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:42.653920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:42.653920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.654193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:42.654193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:42.654193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:42.654193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:42.654193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:42.654193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:42.654193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:42.654193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:42.654193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:42.654193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:42.654193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:42.654193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:42.654193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:42.654410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.654410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:42.654757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.654757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:42.654757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:42.654757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:42.654757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:42.654757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:42.654757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.654757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:42.654757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:42.654757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.654757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:42.654757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:42.654757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.654757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:42.654757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:42.654757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.655294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:42.655294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:42.655294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.655294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:42.655294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:42.655294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:42.655294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.655294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:42.655294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:42.655294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:42.655294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:42.655294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:42.655294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:42.655294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:42.655294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:42.655294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:42.655294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:42.655294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:42.655294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:42.655988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:42.655988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.655988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:42.656594 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:42.656802 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:42.656892 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:42.656987 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:47:42.657359 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:42.657578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.658072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:42.658072 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:42.658072 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:42.658072 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:42.658072 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:42.658072 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:42.658072 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.658072 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:42.658696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.658696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:42.658696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:42.658696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:42.658696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.658696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:42.658696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:42.658696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:42.658696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.658696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:42.658696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:42.658696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:42.658696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:42.658696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:42.658696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:42.658940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:42.658940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:42.658940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:42.658940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:42.658940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:42.658940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:42.658940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:42.658940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.658940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:42.659266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.659266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:42.659266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:42.659266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:42.659266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.659266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:42.659266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:42.659266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:47:42.659266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:42.659266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:42.659266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.659266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:42.659536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:42.659536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:42.659536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.659536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:42.659536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:42.659536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:42.659536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:42.659536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:42.659536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:42.659536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:42.659536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:42.659536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:42.659536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:42.659536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:42.659845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:42.659845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:42.659845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.659845 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:42.660165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.660165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:42.660165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:42.660165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:42.660165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:42.660165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:42.660165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.660165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:42.660165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:42.660165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.660165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:42.660165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:42.660165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.660165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:42.660165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:42.660165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.660806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:42.660806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:42.660806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:47:42.660806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:42.660806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:42.660806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.660806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:42.660806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:42.660806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:42.660806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.660806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:42.660806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:42.660806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:42.660806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:42.660806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:42.660806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:42.660806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:42.660806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:42.661147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:42.661147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:42.661147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:42.661147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:42.661147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:42.661147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.661147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:42.662040 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:42.662219 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:42.662325 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:42.662471 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:42.662747 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:42.662911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.663378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:42.663378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:42.663378 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:42.663378 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:42.663378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:42.663378 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:42.663378 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.663378 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:42.664179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.664179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:42.664179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:42.664179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:42.664179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.664179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:42.664179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:42.664179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:42.664179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.664179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:42.664179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:42.664179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:42.664179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:42.664179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:42.664179 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:42.664424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:42.664424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:42.664424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:42.664424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:42.664424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:42.664424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:42.664424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:42.664424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.664424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:42.665150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.665150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:42.665150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:42.665150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:42.665150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.665150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:42.665150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:42.665150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:42.665150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:42.665150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.665150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:42.665358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:42.665358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:42.665358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.665358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:42.665358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:42.665358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:42.665358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:42.665358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:42.665358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:42.665358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:42.665358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:42.665358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:42.665358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:42.665358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:42.665578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:42.665578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:42.665578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.665578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:42.666312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.666312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:42.666312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:42.666312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:42.666312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:42.666312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:42.666312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.666312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:42.666312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:42.666312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.666312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:42.666312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:42.666312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.666312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:42.666312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:42.666312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.666312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:42.666593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:42.666593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:42.666593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:42.666593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.666593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:42.666593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:42.666593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:42.666593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.666593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:42.666593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:42.666593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:42.666593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:42.666593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:42.666593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:42.666593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:42.666593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:42.666593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:42.666593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:42.666593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:42.666882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:42.666882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:42.666882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.666882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:42.671014 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:42.671323 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:42.671445 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:42.671630 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:42.672019 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:42.672262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:42.672449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.673007 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:42.673007 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:42.673007 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:42.673007 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:42.673007 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:42.673007 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:42.673007 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.673007 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:42.673690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.673690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:42.673690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:42.673690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:42.673690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.673690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:42.673690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:42.673690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.673690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:42.673690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:42.673690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:42.673690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.673888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:42.673888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:42.673888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:42.673888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:42.673888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:42.673888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:42.673888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:42.673888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:42.673888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:42.673888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:42.673888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:42.673888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:42.673888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:42.674070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.674070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:42.674421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.674421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:42.674421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:42.674421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:42.674421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:42.674421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:42.674421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.674421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:42.674421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:42.674421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.674421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:42.674421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:42.674421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.674421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:42.674421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:42.674421 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:42.674803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:42.674803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:42.674803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.674803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:42.674803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:42.674803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:42.674803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:42.674803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:42.674803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:42.674803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:42.674803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:42.674803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:42.674803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:42.674803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:42.674803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:42.674803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:42.674803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:42.674803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:42.674803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:42.675132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:42.675132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:42.675132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:43.854354 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:43.854946 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:43.854946 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:43.855426 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:43.880634 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:43.880798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:43.881222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:43.881222 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:43.881222 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:43.881222 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:43.881222 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:43.881222 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:43.881222 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:43.881222 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:43.881868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:43.881868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:43.881868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:43.881868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:43.881868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:43.881868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:43.881868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:43.881868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:43.881868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:43.881868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:43.881868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:43.881868 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:43.882210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:43.882210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:43.882210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:43.882210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:43.882210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:43.882210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:43.882210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:43.882210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:43.882210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:43.882210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:43.882210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:43.882210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:43.882210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:43.882535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:43.882535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:43.882627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:43.882627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:43.882627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:43.882627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:43.882627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:43.882627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:43.882627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:43.882627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:43.882627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:43.882627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:43.882627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:43.882627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:43.882627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:43.882627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:43.882627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:43.882627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:43.883013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:43.883013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:43.883013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:43.883013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:43.883013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:43.883013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:43.883013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:43.883013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:43.883013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:43.883013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:43.883013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:43.883013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:43.883013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:43.883013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:43.883013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:43.883013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:43.883013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:43.883013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:43.883013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:43.883496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:43.883496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:43.883496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:47.652038 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:47.652717 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:47.652717 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:47.653172 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:47.653476 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:47.653699 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:47.654247 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.654416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:47.654416 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:47.654416 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:47.654416 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:47.654416 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:47.654416 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:47.654416 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.654416 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:47.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:47.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:47.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:47.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:47.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:47.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:47.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:47.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:47.655519 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.655858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:47.655858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:47.655858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:47.655858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:47.655858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:47.655858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:47.655858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:47.655858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:47.655858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:47.655858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:47.655858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:47.655858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:47.655858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:47.656679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.656679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:47.657111 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:47.658414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.658414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:47.658414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:47.658414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:47.658414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:47.658414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:47.658414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.658414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:47.658414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:47.658414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.658414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:47.658414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:47.658414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.658414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:47.658414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:47.658414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:47.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:47.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:47.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:47.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:47.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:47.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:47.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:47.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:47.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:47.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:47.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:47.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:47.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:47.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:47.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:47.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:47.661987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:47.661987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.661987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:47.661987 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:47.661987 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:47.661987 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:47:47.661987 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:47.661987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.662340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:47.662340 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:47.662340 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:47.662340 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:47.662340 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:47.662340 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:47.662340 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.662340 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:47.662340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.662340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:47.662340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:47.662340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:47.662340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.662340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:47.662340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:47.662340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:47.662340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.662340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:47.662677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:47.662677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:47.662677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:47.662677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:47.662677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:47.662677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:47.662677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:47.662677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:47.662677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:47.662677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:47.662677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:47.662677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:47.662677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.662677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:47.662677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.662952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:47.662952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:47.662952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:47.662952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.662952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:47.662952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:47.662952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:47:47.662952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:47.662952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:47.662952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.662952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:47.662952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:47.662952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:47.663141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.663141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:47.663141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:47.663141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:47.663141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:47.663141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:47.663141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:47.663141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:47.663141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:47.663141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:47.663141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:47.663141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:47.663141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:47.663141 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:47.663440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.663440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:47.663440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.663440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:47.663440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:47.663440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:47.663440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:47.663440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:47.663440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.663440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:47.663440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:47.663440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.663440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:47.663440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:47.663440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.663713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:47.663713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:47.663713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.663713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:47.663713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:47.663713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:47:47.663713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:47.663713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:47.663713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.663713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:47.663713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:47.663713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:47.663713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.663713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:47.663713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:47.663713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:47.663713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:47.663713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:47.663713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:47.664027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:47.664027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:47.664027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:47.664027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:47.664027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:47.664027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:47.664027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:47.664027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.664027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:47.664027 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:47.665426 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:47.665426 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:47.665426 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:47.665692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:47.665901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.666840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:47.666840 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:47.666840 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:47.666840 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:47.666840 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:47.666840 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:47.666840 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.666840 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:47.667387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.667387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:47.667387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:47.667387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:47.667387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.667387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:47.667387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:47.667387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:47.667387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.667387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:47.667387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:47.667387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:47.667387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:47.667387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:47.667387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:47.667998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:47.667998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:47.667998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:47.667998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:47.667998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:47.667998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:47.667998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:47.667998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.667998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:47.669043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.669043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:47.669043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:47.669043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:47.669043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.669043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:47.669043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:47.669043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:47.669043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:47.669043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.669043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:47.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:47.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:47.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:47.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:47.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:47.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:47.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:47.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:47.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:47.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:47.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:47.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:47.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:47.671204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:47.671204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:47.671204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.671204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:47.671204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.671204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:47.671204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:47.671204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:47.671204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:47.671204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:47.671204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.671204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:47.671204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:47.671204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.671521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:47.671521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:47.671521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.671521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:47.671521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:47.671521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.671521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:47.671521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:47.671521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:47.671521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:47.671521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.671521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:47.671521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:47.671521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:47.671521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.671521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:47.671521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:47.671521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:47.671521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:47.671851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:47.671851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:47.671851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:47.671851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:47.671851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:47.671851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:47.671851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:47.671851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:47.671851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:47.671851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.671851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:47.671851 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:47.673422 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:47.673422 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:47.673422 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:47.673628 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:47.673829 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:47.674034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.676288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:47.676288 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:47.676288 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:47.676288 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:47.676288 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:47.676288 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:47.676288 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.676288 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:47.676288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.676288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:47.676288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:47.676288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:47.676288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.676288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:47.676288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:47.676633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.676633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:47.676633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:47.676633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:47.676633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.676633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:47.676633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:47.676633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:47.676633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:47.676633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:47.676633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:47.676633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:47.676633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:47.676633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:47.676922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:47.676922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:47.676922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:47.676922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:47.676922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.676922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:47.676922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.676922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:47.676922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:47.676922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:47.676922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:47.676922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:47.676922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.676922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:47.677189 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:47.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:47.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:47.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:47.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:47.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:47.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:47.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:47.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:47.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:47.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:48.854752 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:48.856840 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:48.856840 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:48.856840 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:48.885493 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:48.885692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:48.887464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:48.887464 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:48.887464 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:48.887464 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:48.887464 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:48.887464 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:48.887464 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:48.887464 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:48.887464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:48.887464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:48.887464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:48.887464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:48.887464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:48.887464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:48.887464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:48.888039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:48.888039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:48.888039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:48.888039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:48.888039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:48.888039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:48.888039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:48.888039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:48.888039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:48.888039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:48.888039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:48.888039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:48.888039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:48.888039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:48.888504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:48.888504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:48.888504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:48.888504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:48.888504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:48.888504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:48.888504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:48.888504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:48.888504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:48.888504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:48.888504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:48.888504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:48.888504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:48.888504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:48.888961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:48.889705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:48.889705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:48.889705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:48.889705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:48.889705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:48.889705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:48.889705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:48.889705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:48.889705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:52.652598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:52.653965 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:52.653965 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:52.654300 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:52.654534 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:52.654682 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:52.654797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.655310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:52.655310 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:52.655310 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:52.655310 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:52.655310 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:52.655310 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:52.655310 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.655310 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:52.657914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.657914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:52.657914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:52.657914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:52.657914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.657914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:52.657914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:52.657914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.657914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:52.657914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:52.657914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:52.657914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.658198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:52.658198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:52.658198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:52.658198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:52.658198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:52.658198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:52.658198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:52.658198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:52.658198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:52.658198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:52.658198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:52.658198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:52.658198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:52.658577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.658577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:52.658577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.658577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:52.658577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:52.658577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:52.658577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:52.658577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:52.658577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.658577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:52.658577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:52.658577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.658577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:52.658577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:52.658577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.658954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:52.658954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:52.658954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.658954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:52.658954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:52.658954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.658954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:52.658954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:52.658954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:52.658954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.658954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:52.658954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:52.658954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:52.658954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:52.658954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:52.658954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:52.658954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:52.658954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:52.658954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:52.659447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:52.659447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:52.659447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:52.659447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:52.659447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.659447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:52.659447 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:52.660522 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:52.660522 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:52.660522 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:47:52.660819 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:52.661008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.664279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:52.664279 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:52.664279 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:52.664279 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:52.664279 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:52.664279 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:52.664279 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.664279 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:52.664279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.664279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:52.664279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:52.664279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:52.664279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.664279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:52.664279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:52.664279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:52.664279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.664279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:52.664802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:52.664802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:52.664802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:52.664802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:52.664802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:52.664802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:52.664802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:52.664802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:52.664802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:52.664802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:52.664802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:52.664802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:52.664802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.664802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:52.664802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.665184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:52.665184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:52.665184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:52.665184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.665184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:52.665184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:52.665184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:47:52.665184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:52.665184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:52.665184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.665184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:52.665184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:52.665184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:52.665531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.665531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:52.665531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:52.665531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:52.665531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:52.665531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:52.665531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:52.665531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:52.665531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:52.665531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:52.665531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:52.665531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:52.665531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:52.665531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:52.665893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.665893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:52.666852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.666852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:52.666852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:52.666852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:52.666852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:52.666852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:52.666852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.666852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:52.666852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:52.666852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.666852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:52.666852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:52.666852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.666852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:52.666852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:52.666852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.668409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:52.668409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:52.668409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:47:52.668409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:52.668409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:52.668409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.668409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:52.668409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:52.668409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:52.668409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.668409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:52.668409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:52.668409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:52.668409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:52.668409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:52.668409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:52.668409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:52.668409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:52.670051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:52.670051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:52.670051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:52.670051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:52.670051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:52.670051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.670051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:52.670051 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:52.670051 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:52.670051 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:52.670345 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:52.670345 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:52.670345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.670345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:52.670345 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:52.670345 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:52.670345 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:52.670345 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:52.670345 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:52.670345 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.670345 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:52.672779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.672779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:52.672779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:52.672779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:52.672779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.672779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:52.672779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:52.672779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:52.672779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.672779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:52.672779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:52.672779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:52.672779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:52.672779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:52.672779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:52.673588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:52.673588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:52.673588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:52.673588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:52.673588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:52.673588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:52.673588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:52.673588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.673588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:52.673588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.673588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:52.673588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:52.673588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:52.673588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.674752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:52.674752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:52.674752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:52.674752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:52.674752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.674752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:52.674752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:52.674752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:52.674752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.674752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:52.674752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:52.674752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:52.674752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:52.675058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:52.675058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:52.675058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:52.675058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:52.675058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:52.675058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:52.675058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:52.675058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:52.675058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:52.675058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.675058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:52.675058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.675349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:52.675349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:52.675349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:52.675349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:52.675349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:52.675349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.675349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:52.675349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:52.675349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.675349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:52.675349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:52.675349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.675349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:52.675349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:52.675349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.675349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:52.675349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:52.675349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:52.675349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:52.676243 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:52.677587 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:52.677587 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:52.677587 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:52.677955 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:52.678156 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:52.678304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.680835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:52.680835 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:52.680835 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:52.680835 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:52.680835 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:52.680835 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:52.680835 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.680835 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:52.680835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.680835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:52.680835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:52.680835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:52.680835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.680835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:52.680835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:52.681154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.681154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:52.681154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:52.681154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:52.681154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.681154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:52.681154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:52.681154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:52.681154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:52.681154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:52.681154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:52.681154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:52.681154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:52.681154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:52.681383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:52.681383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:52.681383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:52.681383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:52.681383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.681383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:52.681383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.681383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:52.681383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:52.681383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:52.681383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:52.681383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:52.681383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.681383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:52.681589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:52.681889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:52.681889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:52.681889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:52.681889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:52.681889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:52.681889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:52.681889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:52.681889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:52.681889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:52.681889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:53.855620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:53.856157 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:53.856157 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:53.857207 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:53.867918 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=242 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:47:53.888136 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:53.888501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:53.889244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:53.889244 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:53.889244 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:53.889244 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:53.889244 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:53.889244 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:53.889244 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:53.889244 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:53.890368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:53.890368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:53.890368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:53.890368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:53.890368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:53.890368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:53.890368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:53.890368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:53.890368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:53.890368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:53.890368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:53.890368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:53.890715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:53.890715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:53.890715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:53.890715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:53.890715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:53.890715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:53.890715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:53.890715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:53.890715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:53.890715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:53.890715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:53.890715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:53.890715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:53.891005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:53.891005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:53.891515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:53.891515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:53.891515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:53.891515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:53.891515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:53.891515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:53.891515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:53.891515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:53.891515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:53.891515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:53.891515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:53.891515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:53.891515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:53.891515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:53.891515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:53.891515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:53.892693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:53.892693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:53.892693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:53.892693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:53.892693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:53.892693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:53.892693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:53.892693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:53.892693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:53.892693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:53.892693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:53.892693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:53.892693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:53.892693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:53.892693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:53.892693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:53.892693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:53.892693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:53.892693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:53.893216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:53.893216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:53.893216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:57.653675 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:57.654475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:57.654475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:57.654475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:57.654770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:57.654770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:57.654909 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.655280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:57.655280 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:57.655280 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:57.655280 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:57.655280 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:57.655280 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:57.655280 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.655280 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:57.655899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.655899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:57.655899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:57.655899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:57.655899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.655899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:57.655899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:57.655899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.655899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:57.655899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:57.655899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:57.655899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.656281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:57.656281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:57.656281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:57.656281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:57.656281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:57.656281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:57.656281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:57.656281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:57.656281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:57.656281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:57.656281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:57.656281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:57.656281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:57.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:57.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:57.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:57.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:57.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:57.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:57.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:57.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:57.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:57.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:57.656899 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:57.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:57.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:47:57.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:57.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:57.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:57.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:57.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:57.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:57.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:57.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:57.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:57.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:57.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:57.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:57.657522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:57.658000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:57.658000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:57.658000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:57.658000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:57.658000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.658000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:57.660266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:57.660430 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:57.660524 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:57.660665 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:47:57.660931 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:57.661067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.661458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:57.661458 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:57.661458 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:57.661458 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:57.661458 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:57.661458 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:57.661458 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.661458 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:57.661974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.661974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:57.661974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:57.661974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:57.661974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.661974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:57.661974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:57.661974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:57.661974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.661974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:57.661974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:57.661974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:57.661974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:57.661974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:57.661974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:57.662238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:57.662238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:57.662238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:57.662238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:57.662238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:57.662238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:57.662238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:57.662238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.662238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:57.662580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.662580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:57.662580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:57.662580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:57.662580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.662580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:57.662580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:57.662580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:47:57.662580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:57.662580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:57.662580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.662580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:57.662808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:57.662808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:57.662808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.662808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:57.662808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:57.662808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:57.662808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:57.662808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:57.662808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:57.662808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:57.662808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:57.662808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:57.662808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:57.662808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:57.663030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:57.663030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:57.663030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.663030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:57.663324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.663324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:57.663324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:57.663324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:57.663324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:57.663324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:57.663324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.663324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:57.663324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:57.663324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.663324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:57.663324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:57.663324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.663324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:57.663324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:57.663324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.663678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:47:57.663678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:47:57.663678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:47:57.663678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:57.663678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:57.663678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.663678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:57.663678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:57.663678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:57.663678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.663678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:57.663678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:57.663678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:57.663678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:57.663678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:57.663678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:57.663678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:57.663678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:57.664092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:57.664092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:57.664092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:57.664092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:57.664092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:57.664092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.664092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:57.667991 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:57.668136 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:57.668218 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:57.668357 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:57.668637 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:57.668867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.669261 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:57.669261 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:57.669261 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:57.669261 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:57.669261 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:57.669261 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:57.669261 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.669261 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:57.669806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.669806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:47:57.669806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:57.669806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:57.669806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.669806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:57.669806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:57.669806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:47:57.669806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.669806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:47:57.669806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:57.669806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:47:57.669806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:57.669806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:47:57.669806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:47:57.670115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:57.670115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:47:57.670115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:57.670115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:47:57.670115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:47:57.670115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:57.670115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:47:57.670115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.670115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:47:57.670514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.670514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:57.670514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:57.670514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:57.670514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.670514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:57.670514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:57.670514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:57.670514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:57.670514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.670514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:57.670884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:57.670884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:57.670884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.670884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:57.670884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:57.670884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:57.670884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:57.670884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:57.670884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:57.670884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:57.670884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:57.670884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:57.670884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:57.670884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:57.671293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:57.671293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:57.671293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.671293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:57.671488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.671488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:57.671488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:57.671488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:57.671488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:57.671488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:57.671488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.671488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:57.671488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:57.671488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.671488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:57.671488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:57.671488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.671488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:57.671488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:57.671488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.671488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:47:57.672005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:47:57.672005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:47:57.672005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:47:57.672005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.672005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:57.672005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:57.672005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:57.672005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.672005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:57.672005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:57.672005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:57.672005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:57.672005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:57.672005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:57.672005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:57.672005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:57.672005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:57.672005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:57.672005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:57.672429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:57.672429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:57.672429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.672429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:57.677235 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:57.677422 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:57.677555 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:57.677736 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:57.678034 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:57.678297 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:57.678500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.678952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:57.678952 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:57.678952 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:57.678952 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:57.678952 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:57.678952 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:57.678952 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.678952 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:57.679634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.679634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:57.679634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:57.679634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:57.679634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.679634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:57.679634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:57.679634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.679634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:57.679634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:57.679634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:57.679634 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.680021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:57.680021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:57.680021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:57.680021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:57.680021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:57.680021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:57.680021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:57.680021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:57.680021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:57.680021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:57.680021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:57.680021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:57.680021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:57.680418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.680418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:57.680418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.680418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:57.680418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:57.680418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:57.680418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:57.680418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:57.680418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.680418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:57.680418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:57.680418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.680418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:57.680418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:57.680418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.680957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:57.680957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:57.680957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:57.680957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:47:57.680957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:57.680957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.680957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:57.680957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:57.680957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:57.680957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:57.680957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:57.680957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:57.680957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:57.680957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:57.680957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:57.680957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:57.680957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:57.680957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:57.680957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:57.681460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:57.681460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:57.681460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:57.681460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:57.681460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:57.681460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:47:58.856054 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:47:58.856533 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:47:58.856533 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:47:58.857191 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:47:58.884350 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:47:58.884677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:58.885328 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:47:58.885328 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:58.885328 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:47:58.885328 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:47:58.885328 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:58.885328 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:47:58.885328 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:58.885328 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:47:58.885998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:58.885998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:47:58.885998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:58.885998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:47:58.885998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:58.885998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:58.885998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:58.885998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:58.885998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:58.885998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:58.885998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:47:58.885998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:58.886389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:47:58.886389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:58.886389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:47:58.886389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:58.886389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:47:58.886389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:47:58.886389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:58.886389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:47:58.886389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:58.886389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:47:58.886389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:47:58.886389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:58.886389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:47:58.886721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:58.886721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:47:58.886721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:58.886721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:47:58.886721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:47:58.886721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:47:58.886721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:47:58.886721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:47:58.886721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:58.886721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:47:58.886721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:47:58.886721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:58.886721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:47:58.886721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:47:58.886721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:47:58.887108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:47:58.887108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:47:58.887108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:47:58.887108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:47:58.887108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:47:58.887108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:58.887108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:58.887108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:47:58.887108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:47:58.887108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:47:58.887108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:47:58.887108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:47:58.887108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:47:58.887108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:47:58.887108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:47:58.887108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:47:58.887108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:47:58.887108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:47:58.887108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:47:58.887587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:47:58.887587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:47:58.887587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:47:58.887587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:47:58.887587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:47:58.887587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:02.654043 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:02.654492 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:02.654492 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:02.655106 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:02.655349 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:02.655479 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:02.655596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.656048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:02.656048 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:02.656048 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:02.656048 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:02.656048 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:02.656048 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:02.656048 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.656048 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:02.656719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.656719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:02.656719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:02.656719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:02.656719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.656719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:02.656719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:02.656719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.656719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:02.656719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:02.656719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:02.656719 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.656922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:02.656922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:02.656922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:02.656922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:02.656922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:02.656922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:02.656922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:02.656922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:02.656922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:02.656922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:02.656922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:02.656922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:02.656922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:02.657137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.657137 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:02.657433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.657433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:02.657433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:02.657433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:02.657433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:02.657433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:02.657433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.657433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:02.657433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:02.657433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.657433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:02.657433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:02.657433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.657433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:02.657433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:02.657433 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.657993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:02.657993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:02.657993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.657993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:02.657993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:02.657993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:02.657993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.657993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:02.657993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:02.657993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:02.657993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:02.657993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:02.657993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:02.657993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:02.657993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:02.657993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:02.657993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:02.657993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:02.657993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:02.658457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:02.658457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.658457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:02.660446 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:02.660615 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:02.660701 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:02.660787 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:48:02.661114 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:02.661296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.661727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:02.661727 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:02.661727 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:02.661727 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:02.661727 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:02.661727 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:02.661727 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.661727 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:02.662358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.662358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:02.662358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:02.662358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:02.662358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.662358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:02.662358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:02.662358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:02.662358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.662358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:02.662358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:02.662358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:02.662358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:02.662358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:02.662358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:02.662591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:02.662591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:02.662591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:02.662591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:02.662591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:02.662591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:02.662591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:02.662591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.662591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:02.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:02.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:02.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:02.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:02.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:02.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:48:02.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:02.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:02.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.663032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:02.663461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:02.663461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:02.663461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.663461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:02.663461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:02.663461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:02.663461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:02.663461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:02.663461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:02.663461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:02.663461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:02.663461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:02.663461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:02.663461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:02.663991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:02.663991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:02.663991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.663991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:02.663991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.663991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:02.663991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:02.663991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:02.663991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:02.663991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:02.663991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.663991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:02.663991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:02.663991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.664198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:02.664198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:02.664198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.664198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:02.664198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:02.664198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.664198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:02.664198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:02.664198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:48:02.664198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:02.664198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:02.664198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.664198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:02.664198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:02.664198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:02.664198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.664198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:02.664198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:02.664198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:02.664491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:02.664491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:02.664491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:02.664491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:02.664491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:02.664491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:02.664491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:02.664491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:02.664491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:02.664491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:02.664491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.664491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:02.668265 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:02.668454 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:02.668527 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:02.668626 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:02.668935 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:02.669140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.669527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:02.669527 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:02.669527 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:02.669527 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:02.669527 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:02.669527 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:02.669527 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.669527 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:02.670101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.670101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:02.670101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:02.670101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:02.670101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.670101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:02.670101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:02.670101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:02.670101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.670101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:02.670101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:02.670101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:02.670101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:02.670101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:02.670101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:02.670361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:02.670361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:02.670361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:02.670361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:02.670361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:02.670361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:02.670361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:02.670361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.670361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:02.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:02.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:02.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:02.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:02.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:02.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:02.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:02.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:02.671188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:02.671188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:02.671188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.671188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:02.671188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:02.671188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:02.671188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:02.671188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:02.671188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:02.671188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:02.671188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:02.671188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:02.671188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:02.671188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:02.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:02.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:02.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:02.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:02.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:02.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:02.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:02.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:02.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:02.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:02.671416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.671622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:02.671622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:02.671622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.671622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:02.671622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:02.671622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.671622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:02.671622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:02.671622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:02.671622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:02.671622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.671622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:02.671622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:02.671622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:02.671622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.671622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:02.671622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:02.671622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:02.671622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:02.671921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:02.671921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:02.671921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:02.671921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:02.671921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:02.671921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:02.671921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:02.671921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:02.671921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:02.671921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.671921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:02.677503 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:02.677693 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:02.677778 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:02.677899 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:02.678212 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:02.678433 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:02.678577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.679029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:02.679029 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:02.679029 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:02.679029 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:02.679029 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:02.679029 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:02.679029 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.679029 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:02.679681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.679681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:02.679681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:02.679681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:02.679681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.679681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:02.679681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:02.679681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.679681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:02.679681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:02.679681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:02.679681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.679931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:02.679931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:02.679931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:02.679931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:02.679931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:02.679931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:02.679931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:02.679931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:02.679931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:02.679931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:02.679931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:02.679931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:02.679931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:02.680204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.680204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:02.680502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.680502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:02.680502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:02.680502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:02.680502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:02.680502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:02.680502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.680502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:02.680502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:02.680502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.680502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:02.680502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:02.680502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.680502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:02.680502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:02.680502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:02.681125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:02.681125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:02.681125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.681125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:02.681125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:02.681125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:02.681125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:02.681125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:02.681125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:02.681125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:02.681125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:02.681125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:02.681125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:02.681125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:02.681125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:02.681125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:02.681125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:02.681125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:02.681125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:02.681593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:02.681593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:02.681593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:03.856363 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:03.856914 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:03.856914 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:03.857357 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:03.884583 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:03.884850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:03.885444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:03.885444 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:03.885444 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:03.885444 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:03.885444 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:03.885444 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:03.885444 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:03.885444 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:03.886238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:03.886238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:03.886238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:03.886238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:03.886238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:03.886238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:03.886238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:03.886238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:03.886238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:03.886238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:03.886238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:03.886238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:03.886586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:03.886586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:03.886586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:03.886586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:03.886586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:03.886586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:03.886586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:03.886586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:03.886586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:03.886586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:03.886586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:03.886586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:03.886586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:03.886822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:03.886822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:03.887180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:03.887180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:03.887180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:03.887180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:03.887180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:03.887180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:03.887180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:03.887180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:03.887180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:03.887180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:03.887180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:03.887180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:03.887180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:03.887180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:03.887180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:03.887180 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:03.887564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:03.887564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:03.887564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:03.887564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:03.887564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:03.887564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:03.887564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:03.887564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:03.887564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:03.887564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:03.887564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:03.887564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:03.887564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:03.887564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:03.887564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:03.887564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:03.887564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:03.887564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:03.887564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:03.887982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:03.887982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:03.887982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:07.654541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:07.655149 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:07.655149 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:07.655774 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:07.656020 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:07.656175 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:07.656351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.656944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:07.656944 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:07.656944 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:07.656944 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:07.656944 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:07.656944 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:07.656944 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.656944 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:07.657792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.657792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:07.657792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:07.657792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:07.657792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.657792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:07.657792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:07.657792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.657792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:07.657792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:07.657792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:07.657792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.658043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:07.658043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:07.658043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:07.658043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:07.658043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:07.658043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:07.658043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:07.658043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:07.658043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:07.658043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:07.658043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:07.658043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:07.658043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:07.658272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.658272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:07.658616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.658616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:07.658616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:07.658616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:07.658616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:07.658616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:07.658616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.658616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:07.658616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:07.658616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.658616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:07.658616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:07.658616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.658616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:07.658616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:07.658616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.658942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:07.658942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:07.658942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.658942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:07.658942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:07.658942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:07.658942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.658942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:07.658942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:07.658942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:07.658942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:07.658942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:07.658942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:07.658942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:07.658942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:07.658942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:07.658942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:07.658942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:07.658942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:07.659235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:07.659235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.659235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:07.660721 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:07.660909 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:07.660978 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:07.661077 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:48:07.661512 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:07.661740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.662273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:07.662273 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:07.662273 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:07.662273 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:07.662273 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:07.662273 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:07.662273 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.662273 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:07.662881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.662881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:07.662881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:07.662881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:07.662881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.662881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:07.662881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:07.662881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:07.662881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.662881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:07.662881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:07.662881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:07.662881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:07.662881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:07.662881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:07.663174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:07.663174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:07.663174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:07.663174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:07.663174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:07.663174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:07.663174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:07.663174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.663174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:07.663567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.663567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:07.663567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:07.663567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:07.663567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.663567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:07.663567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:07.663567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:48:07.663567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:07.663567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:07.663567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.663567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:07.664156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:07.664156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:07.664156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.664156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:07.664156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:07.664156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:07.664156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:07.664156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:07.664156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:07.664156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:07.664156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:07.664156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:07.664156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:07.664156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:07.664450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:07.664450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:07.664450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.664450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:07.664450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.664450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:07.664450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:07.664450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:07.664450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:07.664450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:07.664450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.664450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:07.664450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:07.664450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.664744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:07.664744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:07.664744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.664744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:07.664744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:07.664744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.664744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:07.664744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:07.664744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:48:07.664744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:07.664744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:07.664744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.664744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:07.664744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:07.664744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:07.664744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.664744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:07.664744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:07.664744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:07.665041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:07.665041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:07.665041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:07.665041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:07.665041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:07.665041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:07.665041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:07.665041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:07.665041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:07.665041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:07.665041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.665041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:07.668576 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:07.668846 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:07.668943 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:07.669056 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:07.669478 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:07.669731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.670236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:07.670236 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:07.670236 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:07.670236 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:07.670236 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:07.670236 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:07.670236 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.670236 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:07.670843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.670843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:07.670843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:07.670843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:07.670843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.670843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:07.670843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:07.670843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:07.670843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.670843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:07.670843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:07.670843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:07.670843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:07.670843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:07.670843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:07.671111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:07.671111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:07.671111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:07.671111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:07.671111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:07.671111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:07.671111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:07.671111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.671111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:07.671442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.671442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:07.671442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:07.671442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:07.671442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.671442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:07.671442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:07.671442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:07.671442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:07.671442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.671442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:07.672006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:07.672006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:07.672006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.672006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:07.672006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:07.672006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:07.672006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:07.672006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:07.672006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:07.672006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:07.672006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:07.672006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:07.672006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:07.672006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:07.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:07.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:07.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:07.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:07.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:07.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:07.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:07.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:07.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:07.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:07.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.672611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:07.672611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:07.672611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.672611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:07.672611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:07.672611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.672611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:07.672611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:07.672611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:07.672611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:07.672611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.672611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:07.672611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:07.672611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:07.672611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.672611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:07.672611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:07.672611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:07.672611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:07.672898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:07.672898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:07.672898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:07.672898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:07.672898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:07.672898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:07.672898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:07.672898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:07.672898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:07.672898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.672898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:07.677800 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:07.677975 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:07.678060 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:07.678180 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:07.678616 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:07.678903 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:07.679090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.679639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:07.679639 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:07.679639 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:07.679639 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:07.679639 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:07.679639 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:07.679639 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.679639 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:07.680342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.680342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:07.680342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:07.680342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:07.680342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.680342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:07.680342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:07.680342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.680342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:07.680342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:07.680342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:07.680342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.680545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:07.680545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:07.680545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:07.680545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:07.680545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:07.680545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:07.680545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:07.680545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:07.680545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:07.680545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:07.680545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:07.680545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:07.680545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:07.680775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.680775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:07.681021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.681021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:07.681021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:07.681021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:07.681021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:07.681021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:07.681021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.681021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:07.681021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:07.681021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.681021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:07.681021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:07.681021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.681021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:07.681021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:07.681021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:07.681803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:07.681803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:07.681803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.681803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:07.681803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:07.681803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:07.681803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:07.681803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:07.681803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:07.681803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:07.681803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:07.681803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:07.681803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:07.681803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:07.681803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:07.681803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:07.681803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:07.681803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:07.681803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:07.682159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:07.682159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:07.682159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:08.856772 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:08.857388 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:08.857388 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:08.857841 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:08.885128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:08.885383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:08.885791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:08.885791 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:08.885791 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:08.885791 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:08.885791 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:08.885791 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:08.885791 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:08.885791 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:08.886430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:08.886430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:08.886430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:08.886430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:08.886430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:08.886430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:08.886430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:08.886430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:08.886430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:08.886430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:08.886430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:08.886430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:08.886692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:08.886692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:08.886692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:08.886692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:08.886692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:08.886692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:08.886692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:08.886692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:08.886692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:08.886692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:08.886692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:08.886692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:08.886692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:08.886889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:08.886889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:08.887200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:08.887200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:08.887200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:08.887200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:08.887200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:08.887200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:08.887200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:08.887200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:08.887200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:08.887200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:08.887200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:08.887200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:08.887200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:08.887200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:08.887200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:08.887200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:08.888038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:08.888038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:08.888038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:08.888038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:08.888038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:08.888038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:08.888038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:08.888038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:08.888038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:08.888038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:08.888038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:08.888038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:08.888038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:08.888038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:08.888038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:08.888038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:08.888038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:08.888038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:08.888038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:08.888443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:08.888443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:08.888443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:12.655032 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:12.656289 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:12.656289 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:12.656468 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:12.656762 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:12.656966 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:12.657134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.657811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:12.657811 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:12.657811 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:12.657811 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:12.657811 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:12.657811 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:12.657811 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.657811 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:12.658918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.658918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:12.658918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:12.658918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:12.658918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.658918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:12.658918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:12.658918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.658918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:12.658918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:12.658918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:12.658918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.659171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:12.659171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:12.659171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:12.659171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:12.659171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:12.659171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:12.659171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:12.659171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:12.659171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:12.659171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:12.659171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:12.659171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:12.659171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:12.659406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.659406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:12.660075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.660075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:12.660075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:12.660075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:12.660075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:12.660075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:12.660075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.660075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:12.660075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:12.660075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.660075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:12.660075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:12.660075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.660075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:12.660075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:12.660075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.660392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:12.660392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:12.660392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.660392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:12.660392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:12.660392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:12.660392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.660392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:12.660392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:12.660392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:12.660392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:12.660392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:12.660392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:12.660392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:12.660392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:12.660392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:12.660392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:12.660392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:12.660392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:12.660718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:12.660718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.660718 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:12.661043 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:12.661280 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:12.661423 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:12.661561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:48:12.661957 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:12.662119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.662583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:12.662583 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:12.662583 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:12.662583 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:12.662583 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:12.662583 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:12.662583 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.662583 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:12.663464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.663464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:12.663464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:12.663464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:12.663464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.663464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:12.663464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:12.663464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:12.663464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.663464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:12.663464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:12.663464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:12.663464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:12.663464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:12.663464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:12.664016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:12.664016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:12.664016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:12.664016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:12.664016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:12.664016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:12.664016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:12.664016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.664016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:12.664336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.664336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:12.664336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:12.664336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:12.664336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.664336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:12.664336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:12.664336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:48:12.664336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:12.664336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:12.664336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.664336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:12.664791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:12.664791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:12.664791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.664791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:12.664791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:12.664791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:12.664791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:12.664791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:12.664791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:12.664791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:12.664791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:12.664791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:12.664791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:12.664791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:12.665003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:12.665003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:12.665003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.665003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:12.665003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.665003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:12.665003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:12.665003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:12.665003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:12.665003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:12.665003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.665003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:12.665003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:12.665003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.665244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:12.665244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:12.665244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.665244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:12.665244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:12.665244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.665244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:12.665244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:12.665244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:48:12.665244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:12.665244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:12.665244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.665244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:12.665244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:12.665244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:12.665244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.665244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:12.665244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:12.665244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:12.665540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:12.665540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:12.665540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:12.665540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:12.665540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:12.665540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:12.665540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:12.665540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:12.665540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:12.665540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:12.665540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.665540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:12.669035 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:12.669302 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:12.669414 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:12.669557 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:12.670078 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:12.670406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.671029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:12.671029 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:12.671029 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:12.671029 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:12.671029 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:12.671029 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:12.671029 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.671029 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:12.671752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.671752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:12.671752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:12.671752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:12.671752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.671752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:12.671752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:12.671752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:12.671752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.671752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:12.671752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:12.671752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:12.671752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:12.671752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:12.671752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:12.671996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:12.671996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:12.671996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:12.671996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:12.671996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:12.671996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:12.671996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:12.671996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.671996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:12.672409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.672409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:12.672409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:12.672409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:12.672409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.672409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:12.672409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:12.672409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:12.672409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:12.672409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.672409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:12.672613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:12.672613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:12.672613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.672613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:12.672613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:12.672613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:12.672613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:12.672613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:12.672613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:12.672613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:12.672613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:12.672613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:12.672613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:12.672613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:12.672820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:12.672820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:12.672820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.672820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:12.673170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.673170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:12.673170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:12.673170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:12.673170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:12.673170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:12.673170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.673170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:12.673170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:12.673170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.673170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:12.673170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:12.673170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.673170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:12.673170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:12.673170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.673170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:12.673459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:12.673459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:12.673459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:12.673459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.673459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:12.673459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:12.673459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:12.673459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.673459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:12.673459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:12.673459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:12.673459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:12.673459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:12.673459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:12.673459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:12.673459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:12.673459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:12.673459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:12.673459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:12.673764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:12.673764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:12.673764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.673764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:12.678132 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:12.678379 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:12.678484 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:12.678635 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:12.679199 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:12.679605 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:12.679860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.680432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:12.680432 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:12.680432 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:12.680432 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:12.680432 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:12.680432 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:12.680432 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.680432 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:12.681163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.681163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:12.681163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:12.681163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:12.681163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.681163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:12.681163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:12.681163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.681163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:12.681163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:12.681163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:12.681163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.681392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:12.681392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:12.681392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:12.681392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:12.681392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:12.681392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:12.681392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:12.681392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:12.681392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:12.681392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:12.681392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:12.681392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:12.681392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:12.681578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.681578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:12.681895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.681895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:12.681895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:12.681895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:12.681895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:12.681895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:12.681895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.681895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:12.681895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:12.681895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.681895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:12.681895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:12.681895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.681895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:12.681895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:12.681895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:12.682173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:12.682173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:12.682173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.682173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:12.682173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:12.682173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:12.682173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:12.682173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:12.682173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:12.682173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:12.682173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:12.682173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:12.682173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:12.682173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:12.682173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:12.682173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:12.682173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:12.682173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:12.682173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:12.682476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:12.682476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:12.682476 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:13.857373 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:13.858043 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:13.858043 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:13.858336 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:13.887142 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:13.887410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:13.887770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:13.887770 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:13.887770 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:13.887770 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:13.887770 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:13.887770 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:13.887770 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:13.887770 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:13.888398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:13.888398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:13.888398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:13.888398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:13.888398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:13.888398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:13.888398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:13.888398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:13.888398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:13.888398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:13.888398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:13.888398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:13.888695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:13.888695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:13.888695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:13.888695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:13.888695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:13.888695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:13.888695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:13.888695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:13.888695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:13.888695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:13.888695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:13.888695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:13.888695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:13.888980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:13.888980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:13.889447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:13.889447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:13.889447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:13.889447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:13.889447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:13.889447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:13.889447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:13.889447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:13.889447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:13.889447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:13.889447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:13.889447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:13.889447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:13.889447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:13.889447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:13.889447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:13.890031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:13.890031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:13.890031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:13.890031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:13.890031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:13.890031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:13.890031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:13.890031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:13.890031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:13.890031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:13.890031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:13.890031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:13.890031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:13.890031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:13.890031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:13.890031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:13.890031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:13.890031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:13.890031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:13.890362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:13.890362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:13.890362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:17.655517 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:17.656004 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:17.656004 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:17.656681 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:17.656928 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:17.657080 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:17.657244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.657854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:17.657854 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:17.657854 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:17.657854 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:17.657854 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:17.657854 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:17.657854 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.657854 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:17.658584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.658584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:17.658584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:17.658584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:17.658584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.658584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:17.658584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:17.658584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.658584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:17.658584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:17.658584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:17.658584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.658828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:17.658828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:17.658828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:17.658828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:17.658828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:17.658828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:17.658828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:17.658828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:17.658828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:17.658828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:17.658828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:17.658828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:17.658828 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:17.659073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.659073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:17.659418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.659418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:17.659418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:17.659418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:17.659418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:17.659418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:17.659418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.659418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:17.659418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:17.659418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.659418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:17.659418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:17.659418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.659418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:17.659418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:17.659418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.659702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:17.659702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:17.659702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.659702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:17.659702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:17.659702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:17.659702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.659702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:17.659702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:17.659702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:17.659702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:17.659702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:17.659702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:17.659702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:17.659702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:17.659702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:17.659702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:17.659702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:17.659702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:17.660012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:17.660012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.660012 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:17.661298 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:17.661473 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:17.661580 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:17.661693 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:48:17.664957 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:17.665182 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.665818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:17.665818 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:17.665818 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:17.665818 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:17.665818 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:17.665818 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:17.665818 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.665818 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:17.666413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.666413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:17.666413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:17.666413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:17.666413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.666413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:17.666413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:17.666413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:17.666413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.666413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:17.666413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:17.666413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:17.666413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:17.666413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:17.666413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:17.666696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:17.666696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:17.666696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:17.666696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:17.666696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:17.666696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:17.666696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:17.666696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.666696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:17.667106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.667106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:17.667106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:17.667106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:17.667106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.667106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:17.667106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:17.667106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:48:17.667106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:17.667106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:17.667106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.667106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:17.667695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:17.667695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:17.667695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.667695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:17.667695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:17.667695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:17.667695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:17.667695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:17.667695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:17.667695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:17.667695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:17.667695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:17.667695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:17.667695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:17.667991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:17.667991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:17.667991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.667991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:17.667991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.667991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:17.667991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:17.667991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:17.667991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:17.667991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:17.667991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.667991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:17.667991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:17.667991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.668202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:17.668202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:17.668202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.668202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:17.668202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:17.668202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.668202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:17.668202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:17.668202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:48:17.668202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:17.668202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:17.668202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.668202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:17.668202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:17.668202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:17.668202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.668202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:17.668202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:17.668202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:17.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:17.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:17.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:17.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:17.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:17.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:17.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:17.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:17.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:17.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:17.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.668624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:17.669291 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:17.669500 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:17.669575 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:17.669677 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:17.671786 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:17.672073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.672696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:17.672696 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:17.672696 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:17.672696 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:17.672696 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:17.672696 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:17.672696 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.672696 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:17.673262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.673262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:17.673262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:17.673262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:17.673262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.673262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:17.673262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:17.673262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:17.673262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.673262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:17.673262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:17.673262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:17.673262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:17.673262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:17.673262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:17.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:17.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:17.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:17.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:17.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:17.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:17.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:17.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:17.673954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.673954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:17.673954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:17.673954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:17.673954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.673954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:17.673954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:17.673954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:17.673954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:17.673954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.673954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:17.674165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:17.674165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:17.674165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.674165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:17.674165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:17.674165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:17.674165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:17.674165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:17.674165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:17.674165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:17.674165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:17.674165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:17.674165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:17.674165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:17.674404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:17.674404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:17.674404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.674404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:17.674724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.674724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:17.674724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:17.674724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:17.674724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:17.674724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:17.674724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.674724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:17.674724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:17.674724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.674724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:17.674724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:17.674724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.674724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:17.674724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:17.674724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.674724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:17.675025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:17.675025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:17.675025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:17.675025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.675025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:17.675025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:17.675025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:17.675025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.675025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:17.675025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:17.675025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:17.675025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:17.675025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:17.675025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:17.675025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:17.675025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:17.675025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:17.675025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:17.675025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:17.675324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:17.675324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:17.675324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.675324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:17.678418 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:17.678575 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:17.678675 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:17.678817 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:17.679357 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:17.679630 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:17.679821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.680412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:17.680412 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:17.680412 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:17.680412 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:17.680412 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:17.680412 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:17.680412 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.680412 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:17.681018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.681018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:17.681018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:17.681018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:17.681018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.681018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:17.681018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:17.681018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.681018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:17.681018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:17.681018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:17.681018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.681228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:17.681228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:17.681228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:17.681228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:17.681228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:17.681228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:17.681228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:17.681228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:17.681228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:17.681228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:17.681228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:17.681228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:17.681228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:17.681465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.681465 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:17.681792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.681792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:17.681792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:17.681792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:17.681792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:17.681792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:17.681792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.681792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:17.681792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:17.681792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.681792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:17.681792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:17.681792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.681792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:17.681792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:17.681792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:17.682601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:17.682601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:17.682601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.682601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:17.682601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:17.682601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:17.682601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:17.682601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:17.682601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:17.682601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:17.682601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:17.682601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:17.682601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:17.682601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:17.682601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:17.682601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:17.682601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:17.682601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:17.682601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:17.682887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:17.682887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:17.682887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:18.857790 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:18.858435 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:18.858435 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:18.859367 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:18.895880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:18.896132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:18.896707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:18.896707 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:18.896707 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:18.896707 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:18.896707 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:18.896707 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:18.896707 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:18.896707 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:18.897473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:18.897473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:18.897473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:18.897473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:18.897473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:18.897473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:18.897473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:18.897473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:18.897473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:18.897473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:18.897473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:18.897473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:18.897716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:18.897716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:18.897716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:18.897716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:18.897716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:18.897716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:18.897716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:18.897716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:18.897716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:18.897716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:18.897716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:18.897716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:18.897716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:18.897902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:18.897902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:18.898351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:18.898351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:18.898351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:18.898351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:18.898351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:18.898351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:18.898351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:18.898351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:18.898351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:18.898351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:18.898351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:18.898351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:18.898351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:18.898351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:18.898351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:18.898351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:18.898613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:18.898613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:18.898613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:18.898613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:18.898613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:18.898613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:18.898613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:18.898613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:18.898613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:18.898613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:18.898613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:18.898613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:18.898613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:18.898613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:18.898613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:18.898613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:18.898613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:18.898613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:18.898613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:18.898915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:18.898915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:18.898915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:22.655920 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:22.656492 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:22.656492 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:22.656929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:22.657096 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:22.657231 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:22.657392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.657931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:22.657931 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:22.657931 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:22.657931 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:22.657931 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:22.657931 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:22.657931 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.657931 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:22.658706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.658706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:22.658706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:22.658706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:22.658706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.658706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:22.658706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:22.658706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.658706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:22.658706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:22.658706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:22.658706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.659039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:22.659039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:22.659039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:22.659039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:22.659039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:22.659039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:22.659039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:22.659039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:22.659039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:22.659039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:22.659039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:22.659039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:22.659039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:22.659381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.659381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:22.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:22.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:22.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:22.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:22.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:22.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:22.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:22.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:22.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:22.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:22.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:22.659457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.659847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:22.659847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:22.659847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.659847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:22.659847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:22.659847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:22.659847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.659847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:22.659847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:22.659847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:22.659847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:22.659847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:22.659847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:22.659847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:22.659847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:22.659847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:22.659847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:22.659847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:22.659847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:22.660325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:22.660325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.660325 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:22.661555 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:22.661764 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:22.661856 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:22.661953 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:48:22.662245 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:22.662446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.662888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:22.662888 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:22.662888 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:22.662888 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:22.662888 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:22.662888 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:22.662888 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.662888 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:22.663429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.663429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:22.663429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:22.663429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:22.663429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.663429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:22.663429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:22.663429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:22.663429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.663429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:22.663429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:22.663429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:22.663429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:22.663429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:22.663429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:22.663843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:22.663843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:22.663843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:22.663843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:22.663843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:22.663843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:22.663843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:22.663843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.663843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:22.664071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.664071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:22.664071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:22.664071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:22.664071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.664071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:22.664071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:22.664071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:48:22.664071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:22.664071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:22.664071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.664071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:22.664403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:22.664403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:22.664403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.664403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:22.664403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:22.664403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:22.664403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:22.664403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:22.664403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:22.664403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:22.664403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:22.664403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:22.664403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:22.664403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:22.664773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:22.664773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:22.664773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.664773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:22.664898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.664898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:22.664898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:22.664898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:22.664898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:22.664898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:22.664898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.664898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:22.664898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:22.664898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.664898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:22.664898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:22.664898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.664898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:22.664898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:22.664898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.665305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:22.665305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:22.665305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:48:22.665305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:22.665305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:22.665305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.665305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:22.665305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:22.665305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:22.665305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.665305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:22.665305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:22.665305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:22.665305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:22.665305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:22.665305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:22.665305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:22.665305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:22.665764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:22.665764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:22.665764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:22.665764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:22.665764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:22.665764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.665764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:22.669582 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:22.669783 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:22.669966 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:22.670019 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:22.670325 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:22.670551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.670982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:22.670982 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:22.670982 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:22.670982 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:22.670982 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:22.670982 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:22.670982 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.670982 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:22.671553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.671553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:22.671553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:22.671553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:22.671553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.671553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:22.671553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:22.671553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:22.671553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.671553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:22.671553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:22.671553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:22.671553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:22.671553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:22.671553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:22.671820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:22.671820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:22.671820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:22.671820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:22.671820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:22.671820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:22.671820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:22.671820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.671820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:22.672135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.672135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:22.672135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:22.672135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:22.672135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.672135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:22.672135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:22.672135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:22.672135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:22.672135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.672135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:22.672415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:22.672415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:22.672415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.672415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:22.672415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:22.672415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:22.672415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:22.672415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:22.672415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:22.672415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:22.672415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:22.672415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:22.672415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:22.672415 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:22.672664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:22.672664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:22.672664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.672664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:22.672858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.672858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:22.672858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:22.672858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:22.672858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:22.672858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:22.672858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.672858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:22.672858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:22.672858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.672858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:22.672858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:22.672858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.672858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:22.672858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:22.672858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.672858 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:22.673353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:22.673353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:22.673353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:22.673353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.673353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:22.673353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:22.673353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:22.673353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.673353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:22.673353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:22.673353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:22.673353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:22.673353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:22.673353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:22.673353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:22.673353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:22.673353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:22.673353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:22.673353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:22.673786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:22.673786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:22.673786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.673786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:22.678774 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:22.679014 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:22.679108 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:22.679266 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:22.679598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:22.679835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:22.679985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.680470 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:22.680470 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:22.680470 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:22.680470 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:22.680470 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:22.680470 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:22.680470 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.680470 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:22.681112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.681112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:22.681112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:22.681112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:22.681112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.681112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:22.681112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:22.681112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.681112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:22.681112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:22.681112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:22.681112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.681402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:22.681402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:22.681402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:22.681402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:22.681402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:22.681402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:22.681402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:22.681402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:22.681402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:22.681402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:22.681402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:22.681402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:22.681402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:22.681592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.681592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:22.681919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.681919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:22.681919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:22.681919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:22.681919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:22.681919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:22.681919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.681919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:22.681919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:22.681919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.681919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:22.681919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:22.681919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.681919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:22.681919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:22.681919 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:22.682790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:22.682790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:22.682790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.682790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:22.682790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:22.682790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:22.682790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:22.682790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:22.682790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:22.682790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:22.682790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:22.682790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:22.682790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:22.682790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:22.682790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:22.682790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:22.682790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:22.682790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:22.682790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:22.683159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:22.683159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:22.683159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:23.858192 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:23.858715 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:23.858715 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:23.859148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:23.884316 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:23.884448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:23.886274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:23.886274 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:23.886274 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:23.886274 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:23.886274 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:23.886274 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:23.886274 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:23.886274 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:23.886274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:23.886274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:23.886274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:23.886274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:23.886274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:23.886274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:23.886274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:23.886707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:23.886707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:23.886707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:23.886707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:23.886707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:23.886707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:23.886707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:23.886707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:23.886707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:23.886707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:23.886707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:23.886707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:23.886707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:23.886707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:23.886953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:23.886953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:23.886953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:23.886953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:23.886953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:23.886953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:23.886953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:23.886953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:23.886953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:23.886953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:23.886953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:23.886953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:23.886953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:23.886953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:23.887201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:23.887577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:23.887577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:23.887577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:23.887577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:23.887577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:23.887577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:23.887577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:23.887577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:23.887577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:27.656441 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:27.657221 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:27.657221 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:27.657470 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:27.657669 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:27.657840 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:27.657984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.658675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:27.658675 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:27.658675 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:27.658675 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:27.658675 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:27.658675 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:27.658675 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.658675 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:27.659616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.659616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:27.659616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:27.659616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:27.659616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.659616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:27.659616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:27.659616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.659616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:27.659616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:27.659616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:27.659616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.659897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:27.659897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:27.659897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:27.659897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:27.659897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:27.659897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:27.659897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:27.659897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:27.659897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:27.659897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:27.659897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:27.659897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:27.659897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:27.660118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.660118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:27.660714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.660714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:27.660714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:27.660714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:27.660714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:27.660714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:27.660714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.660714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:27.660714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:27.660714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.660714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:27.660714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:27.660714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.660714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:27.660714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:27.660714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.661038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:27.661038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:27.661038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.661038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:27.661038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:27.661038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:27.661038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.661038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:27.661038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:27.661038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:27.661038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:27.661038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:27.661038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:27.661038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:27.661038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:27.661038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:27.661038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:27.661038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:27.661038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:27.661405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:27.661405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.661405 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:27.661877 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:27.662127 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:27.662293 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:27.662450 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:48:27.662856 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:27.663075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.663661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:27.663661 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:27.663661 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:27.663661 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:27.663661 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:27.663661 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:27.663661 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.663661 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:27.664522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.664522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:27.664522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:27.664522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:27.664522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.664522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:27.664522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:27.664522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:27.664522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.664522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:27.664522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:27.664522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:27.664522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:27.664522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:27.664522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:27.664984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:27.664984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:27.664984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:27.664984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:27.664984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:27.664984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:27.664984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:27.664984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.664984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:27.665502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.665502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:27.665502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:27.665502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:27.665502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.665502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:27.665502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:27.665502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:48:27.665502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:27.665502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:27.665502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.665502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:27.665898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:27.665898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:27.665898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.665898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:27.665898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:27.665898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:27.665898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:27.665898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:27.665898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:27.665898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:27.665898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:27.665898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:27.665898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:27.665898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:27.666297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:27.666297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:27.666297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.666297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:27.666918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.666918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:27.666918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:27.666918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:27.666918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:27.666918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:27.666918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.666918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:27.666918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:27.666918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.666918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:27.666918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:27.666918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.666918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:27.666918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:27.666918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.667404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:27.667404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:27.667404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:48:27.667404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:27.667404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:27.667404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.667404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:27.667404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:27.667404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:27.667404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.667404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:27.667404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:27.667404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:27.667404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:27.667404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:27.667404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:27.667404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:27.667404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:27.667910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:27.667910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:27.667910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:27.667910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:27.667910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:27.667910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.667910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:27.669885 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:27.670137 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:27.670258 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:27.670434 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:27.670805 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:27.671063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.671616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:27.671616 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:27.671616 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:27.671616 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:27.671616 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:27.671616 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:27.671616 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.671616 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:27.672487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.672487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:27.672487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:27.672487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:27.672487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.672487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:27.672487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:27.672487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:27.672487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.672487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:27.672487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:27.672487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:27.672487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:27.672487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:27.672487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:27.672925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:27.672925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:27.672925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:27.672925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:27.672925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:27.672925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:27.672925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:27.672925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.672925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:27.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:27.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:27.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:27.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:27.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:27.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:27.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:27.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:27.673875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:27.673875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:27.673875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.673875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:27.673875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:27.673875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:27.673875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:27.673875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:27.673875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:27.673875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:27.673875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:27.673875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:27.673875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:27.673875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:27.674269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:27.674269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:27.674269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.674269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:27.674813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.674813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:27.674813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:27.674813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:27.674813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:27.674813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:27.674813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.674813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:27.674813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:27.674813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.674813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:27.674813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:27.674813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.674813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:27.674813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:27.674813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.674813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:27.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:27.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:27.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:27.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:27.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:27.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:27.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:27.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:27.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:27.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:27.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:27.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:27.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:27.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:27.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:27.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:27.675298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:27.675788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:27.675788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:27.675788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.675788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:27.679186 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:27.679440 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:27.679522 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:27.679676 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:27.680029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:27.680327 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:27.680477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.680987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:27.680987 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:27.680987 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:27.680987 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:27.680987 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:27.680987 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:27.680987 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.680987 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:27.681689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.681689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:27.681689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:27.681689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:27.681689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.681689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:27.681689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:27.681689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.681689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:27.681689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:27.681689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:27.681689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:27.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:27.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:27.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:27.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:27.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:27.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:27.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:27.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:27.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:27.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:27.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:27.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:27.682409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.682409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:27.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:27.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:27.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:27.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:27.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:27.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:27.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:27.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:27.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:27.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:27.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:27.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:27.682904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:27.682904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:27.682904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.682904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:27.682904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:27.682904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:27.682904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:27.682904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:27.682904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:27.682904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:27.682904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:27.682904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:27.682904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:27.682904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:27.682904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:27.682904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:27.682904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:27.682904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:27.682904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:27.683431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:27.683431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:27.683431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:28.858727 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:28.859218 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:28.859218 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:28.859793 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:28.885382 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:28.885616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:28.886162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:28.886162 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:28.886162 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:28.886162 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:28.886162 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:28.886162 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:28.886162 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:28.886162 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:28.886897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:28.886897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:28.886897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:28.886897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:28.886897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:28.886897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:28.886897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:28.886897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:28.886897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:28.886897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:28.886897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:28.886897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:28.887341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:28.887341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:28.887341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:28.887341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:28.887341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:28.887341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:28.887341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:28.887341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:28.887341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:28.887341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:28.887341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:28.887341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:28.887341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:28.887736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:28.887736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:28.887736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:28.887736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:28.887736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:28.887736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:28.887736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:28.887736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:28.887736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:28.887736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:28.887736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:28.887736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:28.887736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:28.887736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:28.887736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:28.888194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:28.888194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:28.888194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:28.888194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:28.888194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:28.888194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:28.888194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:28.888194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:28.888194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:28.888194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:28.888194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:28.888194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:28.888194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:28.888194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:28.888194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:28.888194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:28.888194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:28.888194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:28.888194 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:28.888717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:28.888717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:28.888717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:28.888717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:28.888717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:28.888717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:32.656835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:32.657335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:32.657335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:32.657694 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:32.657838 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:32.657951 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:32.658047 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.658481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:32.658481 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:32.658481 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:32.658481 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:32.658481 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:32.658481 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:32.658481 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.658481 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:32.659110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.659110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:32.659110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:32.659110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:32.659110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.659110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:32.659110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:32.659110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.659110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:32.659110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:32.659110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:32.659110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.659354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:32.659354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:32.659354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:32.659354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:32.659354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:32.659354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:32.659354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:32.659354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:32.659354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:32.659354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:32.659354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:32.659354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:32.659354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:32.659736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.659736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:32.659885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.659885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:32.659885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:32.659885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:32.659885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:32.659885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:32.659885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.659885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:32.659885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:32.659885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.659885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:32.659885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:32.659885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.659885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:32.659885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:32.659885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.660204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:32.660204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:32.660204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.660204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:32.660204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:32.660204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:32.660204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.660204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:32.660204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:32.660204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:32.660204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:32.660204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:32.660204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:32.660204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:32.660204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:32.660204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:32.660204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:32.660204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:32.660204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:32.660532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:32.660532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.660532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:32.662201 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:32.662360 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:32.662451 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:32.662520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:48:32.662804 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:32.662945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.663272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:32.663272 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:32.663272 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:32.663272 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:32.663272 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:32.663272 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:32.663272 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.663272 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:32.663795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.663795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:32.663795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:32.663795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:32.663795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.663795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:32.663795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:32.663795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:32.663795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.663795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:32.663795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:32.663795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:32.663795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:32.663795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:32.663795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:32.664061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:32.664061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:32.664061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:32.664061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:32.664061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:32.664061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:32.664061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:32.664061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.664061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:32.664398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.664398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:32.664398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:32.664398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:32.664398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.664398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:32.664398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:32.664398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:48:32.664398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:32.664398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:32.664398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.664398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:32.664924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:32.664924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:32.664924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.664924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:32.664924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:32.664924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:32.664924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:32.664924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:32.664924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:32.664924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:32.664924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:32.664924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:32.664924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:32.664924 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:32.665388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:32.665388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:32.665388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.665388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:32.665388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.665388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:32.665388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:32.665388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:32.665388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:32.665388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:32.665388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.665388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:32.665388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:32.665388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.665625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:32.665625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:32.665625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.665625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:32.665625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:32.665625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.665625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:32.665625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:32.665625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:48:32.665625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:32.665625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:32.665625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.665625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:32.665625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:32.665625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:32.665625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.665625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:32.665625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:32.665625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:32.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:32.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:32.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:32.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:32.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:32.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:32.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:32.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:32.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:32.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:32.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.665914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:32.670208 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:32.670352 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:32.670448 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:32.670545 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:32.670814 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:32.670983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.671326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:32.671326 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:32.671326 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:32.671326 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:32.671326 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:32.671326 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:32.671326 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.671326 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:32.671848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.671848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:32.671848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:32.671848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:32.671848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.671848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:32.671848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:32.671848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:32.671848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.671848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:32.671848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:32.671848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:32.671848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:32.671848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:32.671848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:32.672088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:32.672088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:32.672088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:32.672088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:32.672088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:32.672088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:32.672088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:32.672088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.672088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:32.672493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.672493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:32.672493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:32.672493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:32.672493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.672493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:32.672493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:32.672493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:32.672493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:32.672493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.672493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:32.672988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:32.672988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:32.672988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.672988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:32.672988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:32.672988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:32.672988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:32.672988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:32.672988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:32.672988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:32.672988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:32.672988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:32.672988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:32.672988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:32.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:32.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:32.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:32.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:32.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:32.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:32.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:32.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:32.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:32.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:32.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.673620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:32.673620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:32.673620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.673620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:32.673620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:32.673620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.673620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:32.673620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:32.673620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:32.673620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:32.673620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.673620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:32.673620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:32.673620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:32.673620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.673620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:32.673620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:32.673620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:32.673620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:32.673996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:32.673996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:32.673996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:32.673996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:32.673996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:32.673996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:32.673996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:32.673996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:32.673996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:32.673996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.673996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:32.679511 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:32.679655 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:32.679748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:32.679882 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:32.680152 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:32.680332 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:32.680449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.680796 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:32.680796 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:32.680796 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:32.680796 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:32.680796 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:32.680796 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:32.680796 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.680796 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:32.681345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.681345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:32.681345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:32.681345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:32.681345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.681345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:32.681345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:32.681345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.681345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:32.681345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:32.681345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:32.681345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.681531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:32.681531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:32.681531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:32.681531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:32.681531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:32.681531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:32.681531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:32.681531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:32.681531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:32.681531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:32.681531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:32.681531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:32.681531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:32.682114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.682114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:32.682114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.682114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:32.682114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:32.682114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:32.682114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:32.682114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:32.682114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.682114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:32.682114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:32.682114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.682114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:32.682114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:32.682114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:32.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:32.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:32.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:32.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:32.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:32.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:32.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:32.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:32.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:32.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:32.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:32.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:32.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:32.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:32.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:32.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:32.682472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:32.683005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:32.683005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:32.683005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:32.683005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:32.683005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:32.683005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:33.859124 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:33.859844 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:33.859844 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:33.860098 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:33.887775 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:33.887944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:33.888469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:33.888469 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:33.888469 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:33.888469 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:33.888469 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:33.888469 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:33.888469 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:33.888469 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:33.889105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:33.889105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:33.889105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:33.889105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:33.889105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:33.889105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:33.889105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:33.889105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:33.889105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:33.889105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:33.889105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:33.889105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:33.889535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:33.889535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:33.889535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:33.889535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:33.889535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:33.889535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:33.889535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:33.889535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:33.889535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:33.889535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:33.889535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:33.889535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:33.889535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:33.889936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:33.889936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:33.889936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:33.889936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:33.889936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:33.889936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:33.889936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:33.889936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:33.889936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:33.889936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:33.889936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:33.889936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:33.889936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:33.889936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:33.889936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:33.890403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:33.890403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:33.890403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:33.890403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:33.890403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:33.890403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:33.890403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:33.890403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:33.890403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:33.890403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:33.890403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:33.890403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:33.890403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:33.890403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:33.890403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:33.890403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:33.890403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:33.890403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:33.890403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:33.890999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:33.890999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:33.890999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:33.890999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:33.890999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:33.890999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:37.657219 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:37.657758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:37.657758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:37.658657 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:37.659292 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:37.659522 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:37.659766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.660450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:37.660450 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:37.660450 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:37.660450 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:37.660450 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:37.660450 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:37.660450 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.660450 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:37.661122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.661122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:37.661122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:37.661122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:37.661122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.661122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:37.661122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:37.661122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.661122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:37.661122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:37.661122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:37.661122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.661517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:37.661517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:37.661517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:37.661517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:37.661517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:37.661517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:37.661517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:37.661517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:37.661517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:37.661517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:37.661517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:37.661517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:37.661517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:37.661877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.661877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:37.661877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.661877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:37.661877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:37.661877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:37.661877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:37.661877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:37.661877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.661877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:37.661877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:37.661877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.661877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:37.661877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:37.661877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.662376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:37.662376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:37.662376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.662376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:37.662376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:37.662376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.662376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:37.662376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:37.662376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:37.662376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.662376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:37.662376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:37.662376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:37.662376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:37.662376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:37.662376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:37.662376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:37.662376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:37.662376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:37.662889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:37.662889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:37.662889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:37.662889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:37.662889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.662889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:37.663506 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:37.663734 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:37.663934 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:37.663934 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:48:37.664390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:37.665314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.665442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:37.665442 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:37.665442 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:37.665442 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:37.665442 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:37.665442 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:37.665442 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.665442 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:37.666023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.666023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:37.666023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:37.666023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:37.666023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.666023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:37.666023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:37.666023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:37.666023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.666023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:37.666023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:37.666023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:37.666023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:37.666023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:37.666023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:37.666776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:37.666776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:37.666776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:37.666776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:37.666776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:37.666776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:37.666776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:37.666776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.666776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:37.666776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.666776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:37.666776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:37.666776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:37.666776 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.667152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:37.667152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:37.667152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:48:37.667152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:37.667152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:37.667152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.667152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:37.667152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:37.667152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:37.667152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.667152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:37.667152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:37.667152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:37.667537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:37.667537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:37.667537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:37.667537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:37.667537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:37.667537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:37.667537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:37.667537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:37.667537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:37.667537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:37.667537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.667537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:37.667864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.667864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:37.667864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:37.667864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:37.667864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:37.667864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:37.667864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.667864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:37.667864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:37.667864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.667864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:37.667864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:37.667864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.667864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:37.667864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:37.667864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.668552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:37.668552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:37.668552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:48:37.668552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:37.668552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:37.668552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.668552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:37.668552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:37.668552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:37.668552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.668552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:37.668552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:37.668552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:37.668552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:37.668552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:37.668552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:37.668552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:37.668552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:37.669056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:37.669056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:37.669056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:37.669056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:37.669056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:37.669056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.669056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:37.670625 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:37.670939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:37.670939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:37.671025 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:37.671385 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:37.671813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.672518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:37.672518 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:37.672518 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:37.672518 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:37.672518 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:37.672518 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:37.672518 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.672518 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:37.673084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.673084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:37.673084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:37.673084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:37.673084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.673084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:37.673084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:37.673084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:37.673084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.673084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:37.673084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:37.673084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:37.673084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:37.673084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:37.673084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:37.673501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:37.673501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:37.673501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:37.673501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:37.673501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:37.673501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:37.673501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:37.673501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.673501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:37.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:37.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:37.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:37.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:37.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:37.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:37.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:37.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:37.674085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:37.674085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:37.674085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.674085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:37.674085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:37.674085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:37.674085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:37.674085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:37.674085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:37.674085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:37.674085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:37.674085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:37.674085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:37.674085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:37.674483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:37.674483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:37.674483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.674483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:37.674483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.674483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:37.674483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:37.674483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:37.674483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:37.674483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:37.674483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.674483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:37.674483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:37.674483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.674869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:37.674869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:37.674869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.674869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:37.674869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:37.674869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.674869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:37.674869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:37.674869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:37.674869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:37.674869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.674869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:37.674869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:37.674869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:37.674869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.674869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:37.674869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:37.674869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:37.674869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:37.675367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:37.675367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:37.675367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:37.675367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:37.675367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:37.675367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:37.675367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:37.675367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:37.675367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:37.675367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.675367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:37.679813 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:37.680178 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:37.680342 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:37.680499 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:37.680821 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:37.681700 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:37.681700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.681872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:37.681872 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:37.681872 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:37.681872 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:37.681872 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:37.681872 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:37.681872 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.681872 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:37.682684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.682684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:37.682684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:37.682684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:37.682684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.682684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:37.682684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:37.682684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.682684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:37.682684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:37.682684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:37.682684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:37.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:37.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:37.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:37.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:37.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:37.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:37.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:37.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:37.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:37.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:37.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:37.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:37.684024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.684024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:37.684176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.684176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:37.684176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:37.684176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:37.684176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:37.684176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:37.684176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.684176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:37.684176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:37.684176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.684176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:37.684176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:37.684176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.684176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:37.684176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:37.684176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:37.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:37.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:37.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:37.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:37.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:37.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:37.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:37.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:37.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:37.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:37.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:37.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:37.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:37.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:37.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:37.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:37.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:37.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:37.685019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:37.685019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:37.685019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:38.859617 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:38.860203 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:38.860338 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:38.861269 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:38.890016 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:38.890198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:38.890687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:38.890687 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:38.890687 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:38.890687 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:38.890687 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:38.890687 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:38.890687 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:38.890687 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:38.891321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:38.891321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:38.891321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:38.891321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:38.891321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:38.891321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:38.891321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:38.891321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:38.891321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:38.891321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:38.891321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:38.891321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:38.891579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:38.891579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:38.891579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:38.891579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:38.891579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:38.891579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:38.891579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:38.891579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:38.891579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:38.891579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:38.891579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:38.891579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:38.891579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:38.891827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:38.891827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:38.892036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:38.892036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:38.892036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:38.892036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:38.892036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:38.892036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:38.892036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:38.892036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:38.892036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:38.892036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:38.892036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:38.892036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:38.892036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:38.892036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:38.892036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:38.892036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:38.892336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:38.892336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:38.892336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:38.892336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:38.892336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:38.892336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:38.892336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:38.892336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:38.892336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:38.892336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:38.892336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:38.892336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:38.892336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:38.892336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:38.892336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:38.892336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:38.892336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:38.892336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:38.892336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:38.892694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:38.892694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:38.892694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:42.657720 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:42.658052 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:42.658162 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:42.658896 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:42.659120 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:42.659302 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:42.659466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.660058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:42.660058 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:42.660058 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:42.660058 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:42.660058 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:42.660058 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:42.660058 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.660058 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:42.660775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.660775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:42.660775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:42.660775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:42.660775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.660775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:42.660775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:42.660775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.660775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:42.660775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:42.660775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:42.660775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:42.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:42.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:42.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:42.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:42.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:42.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:42.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:42.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:42.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:42.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:42.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:42.660988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:42.661229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.661229 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:42.661507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.661507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:42.661507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:42.661507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:42.661507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:42.661507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:42.661507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.661507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:42.661507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:42.661507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.661507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:42.661507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:42.661507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.661507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:42.661507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:42.661507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.661872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:42.661872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:42.661872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.661872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:42.661872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:42.661872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:42.661872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.661872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:42.661872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:42.661872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:42.661872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:42.661872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:42.661872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:42.661872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:42.661872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:42.661872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:42.661872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:42.661872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:42.661872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:42.662391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:42.662391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.662391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:42.663813 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:42.664107 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:42.664240 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:42.664401 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:48:42.664837 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:42.665056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.665688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:42.665688 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:42.665688 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:42.665688 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:42.665688 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:42.665688 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:42.665688 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.665688 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:42.666604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.666604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:42.666604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:42.666604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:42.666604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.666604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:42.666604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:42.666604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:42.666604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.666604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:42.666604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:42.666604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:42.666604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:42.666604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:42.666604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:42.666897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:42.666897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:42.666897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:42.666897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:42.666897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:42.666897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:42.666897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:42.666897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.666897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:42.667505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.667505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:42.667505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:42.667505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:42.667505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.667505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:42.667505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:42.667505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:48:42.667505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:42.667505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:42.667505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.667505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:42.667742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:42.667742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:42.667742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.667742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:42.667742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:42.667742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:42.667742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:42.667742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:42.667742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:42.667742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:42.667742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:42.667742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:42.667742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:42.667742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:42.667962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:42.667962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:42.667962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.667962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:42.668649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.668649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:42.668649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:42.668649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:42.668649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:42.668649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:42.668649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.668649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:42.668649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:42.668649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.668649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:42.668649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:42.668649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.668649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:42.668649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:42.668649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.668923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:42.668923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:42.668923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:48:42.668923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:42.668923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:42.668923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.668923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:42.668923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:42.668923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:42.668923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.668923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:42.668923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:42.668923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:42.668923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:42.668923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:42.668923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:42.668923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:42.668923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:42.669210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:42.669210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:42.669210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:42.669210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:42.669210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:42.669210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.669210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:42.670966 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:42.671370 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:42.671517 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:42.671681 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:42.672076 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:42.672334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.672905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:42.672905 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:42.672905 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:42.672905 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:42.672905 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:42.672905 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:42.672905 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.672905 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:42.673861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.673861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:42.673861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:42.673861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:42.673861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.673861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:42.673861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:42.673861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:42.673861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.673861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:42.673861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:42.673861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:42.673861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:42.673861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:42.673861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:42.674236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:42.674236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:42.674236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:42.674236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:42.674236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:42.674236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:42.674236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:42.674236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.674236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:42.674851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.674851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:42.674851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:42.674851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:42.674851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.674851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:42.674851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:42.674851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:42.674851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:42.674851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.674851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:42.675061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:42.675061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:42.675061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.675061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:42.675061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:42.675061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:42.675061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:42.675061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:42.675061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:42.675061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:42.675061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:42.675061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:42.675061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:42.675061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:42.675327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:42.675327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:42.675327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.675327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:42.676065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.676065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:42.676065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:42.676065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:42.676065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:42.676065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:42.676065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.676065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:42.676065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:42.676065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.676065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:42.676065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:42.676065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.676065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:42.676065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:42.676065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.676065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:42.676497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:42.676497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:42.676497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:42.676497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.676497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:42.676497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:42.676497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:42.676497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.676497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:42.676497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:42.676497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:42.676497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:42.676497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:42.676497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:42.676497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:42.676497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:42.676497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:42.676497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:42.676497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:42.676890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:42.676890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:42.676890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.676890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:42.680220 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:42.680588 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:42.680739 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:42.680928 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:42.681298 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:42.681551 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:42.681723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.682201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:42.682201 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:42.682201 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:42.682201 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:42.682201 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:42.682201 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:42.682201 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.682201 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:42.683196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.683196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:42.683196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:42.683196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:42.683196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.683196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:42.683196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:42.683196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.683196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:42.683196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:42.683196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:42.683196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.683457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:42.683457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:42.683457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:42.683457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:42.683457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:42.683457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:42.683457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:42.683457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:42.683457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:42.683457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:42.683457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:42.683457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:42.683457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:42.683706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.683706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:42.684290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.684290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:42.684290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:42.684290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:42.684290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:42.684290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:42.684290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.684290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:42.684290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:42.684290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.684290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:42.684290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:42.684290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.684290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:42.684290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:42.684290 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:42.684693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:42.684693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:42.684693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.684693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:42.684693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:42.684693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:42.684693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:42.684693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:42.684693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:42.684693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:42.684693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:42.684693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:42.684693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:42.684693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:42.684693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:42.684693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:42.684693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:42.684693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:42.684693 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:42.684978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:42.684978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:42.684978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:43.860342 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:43.860914 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:43.860955 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:43.861989 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:43.898371 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:43.898689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:43.899203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:43.899203 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:43.899203 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:43.899203 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:43.899203 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:43.899203 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:43.899203 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:43.899203 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:43.899998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:43.899998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:43.899998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:43.899998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:43.899998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:43.899998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:43.899998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:43.899998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:43.899998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:43.899998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:43.899998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:43.899998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:43.900348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:43.900348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:43.900348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:43.900348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:43.900348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:43.900348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:43.900348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:43.900348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:43.900348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:43.900348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:43.900348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:43.900348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:43.900348 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:43.900587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:43.900587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:43.900881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:43.900881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:43.900881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:43.900881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:43.900881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:43.900881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:43.900881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:43.900881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:43.900881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:43.900881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:43.900881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:43.900881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:43.900881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:43.900881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:43.900881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:43.900881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:43.901340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:43.901340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:43.901340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:43.901340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:43.901340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:43.901340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:43.901340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:43.901340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:43.901340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:43.901340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:43.901340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:43.901340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:43.901340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:43.901340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:43.901340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:43.901340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:43.901340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:43.901340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:43.901340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:43.901799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:43.901799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:43.901799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:47.658153 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:47.658745 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:47.658880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:47.659791 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:47.660193 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:47.660424 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:47.660571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.661332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:47.661332 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:47.661332 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:47.661332 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:47.661332 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:47.661332 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:47.661332 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.661332 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:47.662403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.662403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:47.662403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:47.662403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:47.662403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.662403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:47.662403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:47.662403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.662403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:47.662403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:47.662403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:47.662403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.662809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:47.662809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:47.662809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:47.662809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:47.662809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:47.662809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:47.662809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:47.662809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:47.662809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:47.662809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:47.662809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:47.662809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:47.662809 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:47.663169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.663169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:47.663504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.663504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:47.663504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:47.663504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:47.663504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:47.663504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:47.663504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.663504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:47.663504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:47.663504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.663504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:47.663504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:47.663504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.663504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:47.663504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:47.663504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.664002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:47.664002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:47.664002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.664002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:47.664002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:47.664002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:47.664002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.664002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:47.664002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:47.664002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:47.664002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:47.664002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:47.664002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:47.664002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:47.664002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:47.664002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:47.664002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:47.664002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:47.664002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:47.664543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:47.664543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.664543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:47.664543 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:47.664543 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:47.664543 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:47.664738 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:48:47.664983 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:47.665166 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.665622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:47.665622 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:47.665622 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:47.665622 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:47.665622 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:47.665622 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:47.665622 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.665622 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:47.666472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.666472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:47.666472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:47.666472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:47.666472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.666472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:47.666472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:47.666472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:47.666472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.666472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:47.666472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:47.666472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:47.666472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:47.666472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:47.666472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:47.666989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:47.666989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:47.666989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:47.666989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:47.666989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:47.666989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:47.666989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:47.666989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.666989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:47.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:47.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:47.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:47.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:47.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:47.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:48:47.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:47.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:47.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:47.667790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:47.667790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:47.667790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.667790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:47.667790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:47.667790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:47.667790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:47.667790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:47.667790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:47.667790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:47.667790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:47.667790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:47.667790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:47.667790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:47.668160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:47.668160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:47.668160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.668160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:47.668473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.668473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:47.668473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:47.668473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:47.668473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:47.668473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:47.668473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.668473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:47.668473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:47.668473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.668473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:47.668473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:47.668473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.668473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:47.668473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:47.668473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.668936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:47.668936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:47.668936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:48:47.668936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:47.668936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:47.668936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.668936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:47.668936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:47.668936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:47.668936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.668936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:47.668936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:47.668936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:47.668936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:47.668936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:47.668936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:47.668936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:47.668936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:47.669451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:47.669451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:47.669451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:47.669451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:47.669451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:47.669451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.669451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:47.671380 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:47.671682 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:47.671815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:47.672035 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:47.672376 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:47.672627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.673147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:47.673147 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:47.673147 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:47.673147 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:47.673147 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:47.673147 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:47.673147 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.673147 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:47.673986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.673986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:47.673986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:47.673986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:47.673986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.673986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:47.673986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:47.673986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:47.673986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.673986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:47.673986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:47.673986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:47.673986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:47.673986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:47.673986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:47.674458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:47.674458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:47.674458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:47.674458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:47.674458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:47.674458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:47.674458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:47.674458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.674458 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:47.674988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.674988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:47.674988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:47.674988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:47.674988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.674988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:47.674988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:47.674988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:47.674988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:47.674988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.674988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:47.675343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:47.675343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:47.675343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.675343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:47.675343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:47.675343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:47.675343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:47.675343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:47.675343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:47.675343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:47.675343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:47.675343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:47.675343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:47.675343 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:47.675742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:47.675742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:47.675742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.675742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:47.676134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.676134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:47.676134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:47.676134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:47.676134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:47.676134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:47.676134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.676134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:47.676134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:47.676134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.676134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:47.676134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:47.676134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.676134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:47.676134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:47.676134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.676134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:47.676455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:47.676455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:47.676455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:47.676455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.676455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:47.676455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:47.676455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:47.676455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.676455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:47.676455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:47.676455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:47.676455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:47.676455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:47.676455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:47.676455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:47.676455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:47.676455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:47.676455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:47.676455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:47.676754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:47.676754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:47.676754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.676754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:47.680596 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:47.680887 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:47.681021 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:47.681233 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:47.681567 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:47.681798 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:47.681926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.682358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:47.682358 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:47.682358 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:47.682358 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:47.682358 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:47.682358 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:47.682358 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.682358 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:47.683338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.683338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:47.683338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:47.683338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:47.683338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.683338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:47.683338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:47.683338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.683338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:47.683338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:47.683338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:47.683338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.683711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:47.683711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:47.683711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:47.683711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:47.683711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:47.683711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:47.683711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:47.683711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:47.683711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:47.683711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:47.683711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:47.683711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:47.683711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:47.684008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.684008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:47.684396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.684396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:47.684396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:47.684396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:47.684396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:47.684396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:47.684396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.684396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:47.684396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:47.684396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.684396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:47.684396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:47.684396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.684396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:47.684396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:47.684396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:47.684884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:47.684884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:47.684884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.684884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:47.684884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:47.684884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:47.684884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:47.684884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:47.684884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:47.684884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:47.684884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:47.684884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:47.684884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:47.684884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:47.684884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:47.684884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:47.684884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:47.684884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:47.684884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:47.685363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:47.685363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:47.685363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:48.860770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:48.861283 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:48.861452 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:48.862336 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:48.928379 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:48.928691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:48.929466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:48.929466 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:48.929466 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:48.929466 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:48.929466 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:48.929466 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:48.929466 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:48.929466 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:48.930757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.068 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:48.930757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:48.930757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:48.930757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:48.930757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:48.930757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:48.930757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:48.930757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:48.930757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:48.930757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:48.930757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:48.930757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:48.931053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:48.931053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:48.931053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:48.931053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:48.931053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:48.931053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:48.931053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:48.931053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:48.931053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:48.931053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:48.931053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:48.931053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:48.931053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:48.931304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:48.931304 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:48.931968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:48.931968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:48.931968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:48.931968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:48.931968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:48.931968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:48.931968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:48.931968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:48.931968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:48.931968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:48.931968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:48.931968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:48.931968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:48.931968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:48.931968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:48.931968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:48.932350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:48.932350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:48.932350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:48.932350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:48.932350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:48.932350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:48.932350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:48.932350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:48.932350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:48.932350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:48.932350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:48.932350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:48.932350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:48.932350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:48.932350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:48.932350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:48.932350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:48.932350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:48.932350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:48.932736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:48.932736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:48.932736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:52.658830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:52.659455 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:52.659455 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:52.660713 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:52.660713 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:52.660713 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:52.660713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.660835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:52.660835 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:52.660835 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:52.660835 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:52.660835 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:52.660835 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:52.660835 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.660835 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:52.661566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.661566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:52.661566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:52.661566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:52.661566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.661566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:52.661566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:52.661566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.661566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:52.661566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:52.661566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:52.661566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.661836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:52.661836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:52.661836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:52.661836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:52.661836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:52.661836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:52.661836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:52.661836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:52.661836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:52.661836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:52.661836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:52.661836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:52.661836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:52.662082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.662082 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:52.662282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.662282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:52.662282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:52.662282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:52.662282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:52.662282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:52.662282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.662282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:52.662282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:52.662282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.662282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:52.662282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:52.662282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.662282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:52.662282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:52.662282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:52.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:52.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:52.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:52.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:52.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:52.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:52.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:52.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:52.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:52.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:52.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:52.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:52.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:52.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:52.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:52.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:52.662887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:52.662887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.662887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:52.664412 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:52.664717 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:52.664794 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:52.664887 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:48:52.665770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:52.665770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.665910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:52.665910 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:52.665910 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:52.665910 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:52.665910 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:52.665910 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:52.665910 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.665910 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:52.666547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.666547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:52.666547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:52.666547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:52.666547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.666547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:52.666547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:52.666547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:52.666547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.666547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:52.666547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:52.666547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:52.666547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:52.666547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:52.666547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:52.666822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:52.666822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:52.666822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:52.666822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:52.666822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:52.666822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:52.666822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:52.666822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.666822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:52.667176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.667176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:52.667176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:52.667176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:52.667176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.667176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:52.667176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:52.667176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:48:52.667176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:52.667176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:52.667176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.667176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:52.667403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:52.667403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:52.667403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.667403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:52.667403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:52.667403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:52.667403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:52.667403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:52.667403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:52.667403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:52.667403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:52.667403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:52.667403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:52.667403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:52.667618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:52.667618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:52.667618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.667618 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:52.667912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.667912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:52.667912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:52.667912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:52.667912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:52.667912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:52.667912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.667912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:52.667912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:52.667912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.667912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:52.667912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:52.667912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.667912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:52.667912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:52.667912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.668197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:52.668197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:52.668197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:48:52.668197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:52.668197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:52.668197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.668197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:52.668197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:52.668197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:52.668197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.668197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:52.668197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:52.668197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:52.668197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:52.668197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:52.668197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:52.668197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:52.668197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:52.668513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:52.668513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:52.668513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:52.668513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:52.668513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:52.668513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.668513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:52.671828 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:52.672076 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:52.672167 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:52.672330 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:52.673226 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:52.673226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:52.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:52.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:52.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:52.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:52.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:52.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.673377 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:52.674016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.674016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:52.674016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:52.674016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:52.674016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.674016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:52.674016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:52.674016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:52.674016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.674016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:52.674016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:52.674016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:52.674016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:52.674016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:52.674016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:52.674309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:52.674309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:52.674309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:52.674309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:52.674309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:52.674309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:52.674309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:52.674309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.674309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:52.674629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.674629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:52.674629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:52.674629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:52.674629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.674629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:52.674629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:52.674629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:52.674629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:52.674629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.674629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:52.674840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:52.674840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:52.674840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.674840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:52.674840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:52.674840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:52.674840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:52.674840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:52.674840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:52.674840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:52.674840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:52.674840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:52.674840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:52.674840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:52.675061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:52.675061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:52.675061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.675061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:52.675496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.675496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:52.675496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:52.675496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:52.675496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:52.675496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:52.675496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.675496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:52.675496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:52.675496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.675496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:52.675496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:52.675496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.675496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:52.675496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:52.675496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.675496 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:52.675827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:52.675827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:52.675827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:52.675827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.675827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:52.675827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:52.675827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:52.675827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.675827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:52.675827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:52.675827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:52.675827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:52.675827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:52.675827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:52.675827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:52.675827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:52.675827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:52.675827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:52.675827 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:52.676116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:52.676116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:52.676116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.676116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:52.681067 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:52.681307 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:52.681416 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:52.681564 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:52.682625 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:52.682625 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:52.682625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.682764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:52.682764 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:52.682764 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:52.682764 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:52.682764 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:52.682764 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:52.682764 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.682764 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:52.683622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.683622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:52.683622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:52.683622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:52.683622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.683622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:52.683622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:52.683622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.683622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:52.683622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:52.683622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:52.683622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.683886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:52.683886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:52.683886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:52.683886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:52.683886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:52.683886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:52.683886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:52.683886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:52.683886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:52.683886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:52.683886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:52.683886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:52.683886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:52.684100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.684100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:52.684430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.684430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:52.684430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:52.684430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:52.684430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:52.684430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:52.684430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.684430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:52.684430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:52.684430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.684430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:52.684430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:52.684430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.684430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:52.684430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:52.684430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:52.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:52.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:52.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:52.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:52.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:52.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:52.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:52.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:52.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:52.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:52.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:52.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:52.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:52.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:52.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:52.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:52.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:52.684714 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:52.685051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:52.685051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:52.685051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:53.861272 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:53.861745 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:53.861745 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:53.862216 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:53.871511 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=241 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:48:53.889772 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:53.889936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:53.890373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:53.890373 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:53.890373 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:53.890373 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:53.890373 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:53.890373 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:53.890373 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:53.890373 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:53.891108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:53.891108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:53.891108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:53.891108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:53.891108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:53.891108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:53.891108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:53.891108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:53.891108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:53.891108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:53.891108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:53.891108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:53.891374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:53.891374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:53.891374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:53.891374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:53.891374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:53.891374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:53.891374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:53.891374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:53.891374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:53.891374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:53.891374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:53.891374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:53.891374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:53.891631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:53.891631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:53.891926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:53.891926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:53.891926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:53.891926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:53.891926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:53.891926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:53.891926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:53.891926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:53.891926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:53.891926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:53.891926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:53.891926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:53.891926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:53.891926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:53.891926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:53.891926 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:53.892273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:53.892273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:53.892273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:53.892273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:53.892273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:53.892273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:53.892273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:53.892273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:53.892273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:53.892273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:53.892273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:53.892273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:53.892273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:53.892273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:53.892273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:53.892273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:53.892273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:53.892273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:53.892273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:53.892611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:53.892611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:53.892611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:57.659215 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:57.659898 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:57.660074 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:57.660808 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:57.661053 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:57.661204 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:57.661373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.661890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:57.661890 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:57.661890 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:57.661890 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:57.661890 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:57.661890 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:57.661890 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.661890 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:57.662989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.662989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:57.662989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:57.662989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:57.662989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.662989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:57.662989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:57.662989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.662989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:57.662989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:57.662989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:57.662989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.663240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:57.663240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:57.663240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:57.663240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:57.663240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:57.663240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:57.663240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:57.663240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:57.663240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:57.663240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:57.663240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:57.663240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:57.663240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:57.663481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.663481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:57.664171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.664171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:57.664171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:57.664171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:57.664171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:57.664171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:57.664171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.664171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:57.664171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:57.664171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.664171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:57.664171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:57.664171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.664171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:57.664171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:57.664171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.664485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:48:57.664485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:57.664485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.664485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:57.664485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:57.664485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:57.664485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.664485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:57.664485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:57.664485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:57.664485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:57.664485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:57.664485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:57.664485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:57.664485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:57.664485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:57.664485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:57.664485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:57.664485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:57.664890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:57.664890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.664890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:57.664890 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:57.665328 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:57.665328 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:57.665480 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:48:57.665776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:57.665960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.666450 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:57.666450 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:57.666450 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:57.666450 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:57.666450 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:57.666450 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:57.666450 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.666450 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:57.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:57.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:57.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:57.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:57.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:57.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:57.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:57.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:57.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:57.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:57.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:57.667374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:57.667981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:57.667981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:57.667981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:57.667981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:57.667981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:57.667981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:57.667981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:57.667981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.667981 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:57.668301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.668301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:57.668301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:57.668301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:57.668301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.668301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:57.668301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:57.668301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:48:57.668301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:57.668301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:57.668301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.668301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:57.668532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:57.668532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:57.668532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.668532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:57.668532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:57.668532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:57.668532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:57.668532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:57.668532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:57.668532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:57.668532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:57.668532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:57.668532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:57.668532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:57.668770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:57.668770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:57.668770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.668770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:57.669491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.669491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:57.669491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:57.669491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:57.669491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:57.669491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:57.669491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.669491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:57.669491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:57.669491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.669491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:57.669491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:57.669491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.669491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:57.669491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:57.669491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.669846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:48:57.669846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:48:57.669846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:48:57.669846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:57.669846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:57.669846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.669846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:57.669846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:57.669846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:57.669846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.669846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:57.669846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:57.669846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:57.669846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:57.669846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:57.669846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:57.669846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:57.669846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:57.670116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:57.670116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:57.670116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:57.670116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:57.670116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:57.670116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.670116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:57.672185 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:57.672610 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:57.672767 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:57.672929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:57.673325 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:57.673556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.674043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:57.674043 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:57.674043 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:57.674043 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:57.674043 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:57.674043 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:57.674043 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.674043 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:57.674978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.674978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:48:57.674978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:57.674978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:57.674978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.674978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:57.674978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:57.674978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:48:57.674978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.674978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:48:57.674978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:57.674978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:48:57.674978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:57.674978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:48:57.674978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:48:57.675246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:57.675246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:48:57.675246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:57.675246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:48:57.675246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:48:57.675246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:57.675246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:48:57.675246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.675246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:48:57.675902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.675902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:57.675902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:57.675902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:57.675902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.675902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:57.675902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:57.675902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:57.675902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:57.675902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.675902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:57.676106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:57.676106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:57.676106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.676106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:57.676106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:57.676106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:57.676106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:57.676106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:57.676106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:57.676106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:57.676106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:57.676106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:57.676106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:57.676106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:57.676321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:57.676321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:57.676321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.676321 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:57.677110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.677110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:57.677110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:57.677110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:57.677110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:57.677110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:57.677110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.677110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:57.677110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:57.677110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.677110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:57.677110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:57.677110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.677110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:57.677110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:57.677110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.677110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:48:57.677432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:48:57.677432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:48:57.677432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:48:57.677432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.677432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:57.677432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:57.677432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:57.677432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.677432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:57.677432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:57.677432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:57.677432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:57.677432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:57.677432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:57.677432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:57.677432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:57.677432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:57.677432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:57.677432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:57.677773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:57.677773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:57.677773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.677773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:57.681456 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:57.681921 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:57.682062 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:57.682263 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:57.682690 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:57.682894 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:57.683028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.683533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:57.683533 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:57.683533 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:57.683533 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:57.683533 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:57.683533 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:57.683533 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.683533 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:57.684611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.684611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:57.684611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:57.684611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:57.684611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.684611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:57.684611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:57.684611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.684611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:57.684611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:57.684611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:57.684611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.684832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:57.684832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:57.684832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:57.684832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:57.684832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:57.684832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:57.684832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:57.684832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:57.684832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:57.684832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:57.684832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:57.684832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:57.684832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:57.685044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.685044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:57.685805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.685805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:57.685805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:57.685805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:57.685805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:57.685805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:57.685805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.685805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:57.685805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:57.685805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.685805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:57.685805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:57.685805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.685805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:57.685805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:57.685805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:57.686094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:48:57.686094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:57.686094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.686094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:57.686094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:57.686094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:57.686094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:57.686094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:57.686094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:57.686094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:57.686094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:57.686094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:57.686094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:57.686094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:57.686094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:57.686094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:57.686094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:57.686094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:57.686094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:57.686439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:57.686439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:57.686439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:48:58.861788 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:48:58.862406 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:48:58.862406 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:48:58.862823 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:48:58.889835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:48:58.890026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:58.890422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:48:58.890422 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:58.890422 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:48:58.890422 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:48:58.890422 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:58.890422 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:48:58.890422 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:58.890422 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:48:58.891025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:58.891025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:48:58.891025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:58.891025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:48:58.891025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:58.891025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:58.891025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:58.891025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:58.891025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:58.891025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:58.891025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:48:58.891025 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:58.891819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:48:58.891819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:58.891819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:48:58.891819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:58.891819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:48:58.891819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:48:58.891819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:58.891819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:48:58.891819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:58.891819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:48:58.891819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:48:58.891819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:58.891819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:48:58.892652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:58.892652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:48:58.892652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:58.892652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:48:58.892652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:48:58.892652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:48:58.892652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:48:58.892652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:48:58.892652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:58.892652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:48:58.892652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:48:58.892652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:58.892652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:48:58.892652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:48:58.892652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:48:58.893784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:48:58.893784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:48:58.893784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:48:58.893784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:48:58.893784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:48:58.893784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:58.893784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:58.893784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:48:58.893784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:48:58.893784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:48:58.893784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:48:58.893784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:48:58.893784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:48:58.893784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:48:58.893784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:48:58.893784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:48:58.893784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:48:58.893784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:48:58.893784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:48:58.894841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:48:58.894841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:48:58.894841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:48:58.894841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:48:58.894841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:48:58.894841 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:02.659906 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:02.660552 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:02.660552 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:02.660930 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:02.661109 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:02.661268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:02.661406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.661891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:02.661891 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:02.661891 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:02.661891 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:02.661891 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:02.661891 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:02.661891 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.661891 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:02.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:02.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:02.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:02.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:02.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:02.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:02.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:02.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:02.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.663173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:02.663173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:02.663173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:02.663173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:02.663173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:02.663173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:02.663173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:02.663173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:02.663173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:02.663173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:02.663173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:02.663173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:02.663173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:02.663734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.663734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:02.663734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.663734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:02.663734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:02.663734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:02.663734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:02.663734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:02.663734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.663734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:02.663734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:02.663734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.663734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:02.663734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:02.663734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.664083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:02.664083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:02.664083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.664083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:02.664083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:02.664083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.664083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:02.664083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:02.664083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:02.664083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.664083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:02.664083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:02.664083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:02.664083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:02.664083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:02.664083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:02.664083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:02.664083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:02.664083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:02.664676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:02.664676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:02.664676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:02.664676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:02.664676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.664676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:02.665319 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:02.665461 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:02.665566 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:02.665675 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:49:02.665935 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:02.666086 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.666475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:02.666475 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:02.666475 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:02.666475 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:02.666475 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:02.666475 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:02.666475 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.666475 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:02.666979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.666979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:02.666979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:02.666979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:02.666979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.666979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:02.666979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:02.666979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:02.666979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.666979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:02.666979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:02.666979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:02.666979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:02.666979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:02.666979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:02.667579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:02.667579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:02.667579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:02.667579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:02.667579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:02.667579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:02.667579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:02.667579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.667579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:02.667579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.667579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:02.667579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:02.667579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:02.667579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:02.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:02.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:49:02.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:02.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:02.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:02.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:02.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:02.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:02.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:02.668332 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:02.668979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:02.668979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:02.668979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:02.668979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:02.668979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:02.668979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:02.668979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:02.668979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:02.668979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:02.668979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:02.668979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.668979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:02.669568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.669568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:02.669568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:02.669568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:02.669568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:02.669568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:02.669568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.669568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:02.669568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:02.669568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.669568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:02.669568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:02.669568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.669568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:02.669568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:02.669568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.669975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:02.669975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:02.669975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:49:02.669975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:02.669975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:02.669975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.669975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:02.669975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:02.669975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:02.669975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.669975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:02.669975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:02.669975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:02.669975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:02.669975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:02.669975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:02.669975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:02.669975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:02.670441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:02.670441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:02.670441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:02.670441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:02.670441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:02.670441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.670441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:02.672732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:02.673090 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:02.673141 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:02.673237 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:02.673539 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:02.673715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.674045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:02.674045 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:02.674045 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:02.674045 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:02.674045 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:02.674045 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:02.674045 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.674045 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:02.674559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.674559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:02.674559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:02.674559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:02.674559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.674559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:02.674559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:02.674559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:02.674559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.674559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:02.674559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:02.674559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:02.674559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:02.674559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:02.674559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:02.674966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:02.674966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:02.674966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:02.674966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:02.674966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:02.674966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:02.674966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:02.674966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.674966 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:02.675199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.675199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:02.675199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:02.675199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:02.675199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.675199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:02.675199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:02.675199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:02.675199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:02.675199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.675199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:02.675499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:02.675499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:02.675499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.675499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:02.675499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:02.675499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:02.675499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:02.675499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:02.675499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:02.675499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:02.675499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:02.675499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:02.675499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:02.675499 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:02.675829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:02.675829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:02.675829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.675829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:02.675829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.675829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:02.675829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:02.675829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:02.675829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:02.675829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:02.675829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.675829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:02.675829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:02.675829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.676096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:02.676096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:02.676096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.676096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:02.676096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:02.676096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.676096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:02.676096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:02.676096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:02.676096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:02.676096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.676096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:02.676096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:02.676096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:02.676096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.676096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:02.676096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:02.676096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:02.676096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:02.676480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:02.676480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:02.676480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:02.676480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:02.676480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:02.676480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:02.676480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:02.676480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:02.676480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:02.676480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.676480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:02.681979 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:02.682265 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:02.682380 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:02.682526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:02.682777 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:02.682948 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:02.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.683473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:02.683473 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:02.683473 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:02.683473 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:02.683473 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:02.683473 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:02.683473 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.683473 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:02.684063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.684063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:02.684063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:02.684063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:02.684063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.684063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:02.684063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:02.684063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.684063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:02.684063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:02.684063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:02.684063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.684397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:02.684397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:02.684397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:02.684397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:02.684397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:02.684397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:02.684397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:02.684397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:02.684397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:02.684397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:02.684397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:02.684397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:02.684397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:02.684711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.684711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:02.684768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.684768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:02.684768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:02.684768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:02.684768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:02.684768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:02.684768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.684768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:02.684768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:02.684768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.684768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:02.684768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:02.684768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.684768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:02.684768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:02.684768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:02.685149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:02.685149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:02.685149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.685149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:02.685149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:02.685149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:02.685149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:02.685149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:02.685149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:02.685149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:02.685149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:02.685149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:02.685149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:02.685149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:02.685149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:02.685149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:02.685149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:02.685149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:02.685149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:02.685623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:02.685623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:02.685623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:03.862148 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:03.862835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:03.862835 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:03.863066 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:03.889613 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:03.889888 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:03.890520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:03.890520 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:03.890520 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:03.890520 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:03.890520 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:03.890520 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:03.890520 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:03.890520 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:03.891210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:03.891210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:03.891210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:03.891210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:03.891210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:03.891210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:03.891210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:03.891210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:03.891210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:03.891210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:03.891210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:03.891210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:03.891436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:03.891436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:03.891436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:03.891436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:03.891436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:03.891436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:03.891436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:03.891436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:03.891436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:03.891436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:03.891436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:03.891436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:03.891436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:03.891631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:03.891631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:03.892020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:03.892020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:03.892020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:03.892020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:03.892020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:03.892020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:03.892020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:03.892020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:03.892020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:03.892020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:03.892020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:03.892020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:03.892020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:03.892020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:03.892020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:03.892020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:03.892311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:03.892311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:03.892311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:03.892311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:03.892311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:03.892311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:03.892311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:03.892311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:03.892311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:03.892311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:03.892311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:03.892311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:03.892311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:03.892311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:03.892311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:03.892311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:03.892311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:03.892311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:03.892311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:03.892595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:03.892595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:03.892595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:07.660381 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:07.660880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:07.660880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:07.661449 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:07.661660 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:07.661778 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:07.661894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.662418 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:07.662418 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:07.662418 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:07.662418 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:07.662418 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:07.662418 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:07.662418 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.662418 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:07.663543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.663543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:07.663543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:07.663543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:07.663543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.663543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:07.663543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:07.663543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.663543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:07.663543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:07.663543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:07.663543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.663788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:07.663788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:07.663788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:07.663788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:07.663788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:07.663788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:07.663788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:07.663788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:07.663788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:07.663788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:07.663788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:07.663788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:07.663788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:07.664069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.664069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:07.664754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.664754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:07.664754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:07.664754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:07.664754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:07.664754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:07.664754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.664754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:07.664754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:07.664754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.664754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:07.664754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:07.664754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.664754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:07.664754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:07.664754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.665009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:07.665009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:07.665009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.665009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:07.665009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:07.665009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:07.665009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.665009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:07.665009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:07.665009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:07.665009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:07.665009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:07.665009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:07.665009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:07.665009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:07.665009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:07.665009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:07.665009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:07.665009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:07.665311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:07.665311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.665311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:07.665546 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:07.665828 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:07.665949 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:07.666089 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:49:07.666400 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:07.666549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.666980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:07.666980 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:07.666980 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:07.666980 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:07.666980 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:07.666980 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:07.666980 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.666980 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:07.667882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.667882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:07.667882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:07.667882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:07.667882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.667882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:07.667882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:07.667882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:07.667882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.667882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:07.667882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:07.667882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:07.667882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:07.667882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:07.667882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:07.668423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:07.668423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:07.668423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:07.668423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:07.668423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:07.668423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:07.668423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:07.668423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.668423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:07.668843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.668843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:07.668843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:07.668843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:07.668843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.668843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:07.668843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:07.668843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:49:07.668843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:07.668843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:07.668843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.668843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:07.669031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:07.669031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:07.669031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.669031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:07.669031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:07.669031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:07.669031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:07.669031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:07.669031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:07.669031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:07.669031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:07.669031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:07.669031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:07.669031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:07.669233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:07.669233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:07.669233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.669233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:07.670005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.670005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:07.670005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:07.670005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:07.670005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:07.670005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:07.670005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.670005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:07.670005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:07.670005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.670005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:07.670005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:07.670005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.670005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:07.670005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:07.670005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.670268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:07.670268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:07.670268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:49:07.670268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:07.670268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:07.670268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.670268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:07.670268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:07.670268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:07.670268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.670268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:07.670268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:07.670268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:07.670268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:07.670268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:07.670268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:07.670268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:07.670268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:07.670516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:07.670516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:07.670516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:07.670516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:07.670516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:07.670516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.670516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:07.673153 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:07.673476 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:07.673651 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:07.673777 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:07.674084 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:07.674265 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.674689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:07.674689 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:07.674689 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:07.674689 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:07.674689 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:07.674689 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:07.674689 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.674689 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:07.675615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.675615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:07.675615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:07.675615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:07.675615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.675615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:07.675615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:07.675615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:07.675615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.675615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:07.675615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:07.675615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:07.675615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:07.675615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:07.675615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:07.675918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:07.675918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:07.675918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:07.675918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:07.675918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:07.675918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:07.675918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:07.675918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.675918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:07.676605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.676605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:07.676605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:07.676605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:07.676605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.676605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:07.676605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:07.676605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:07.676605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:07.676605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.676605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:07.676802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:07.676802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:07.676802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.676802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:07.676802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:07.676802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:07.676802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:07.676802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:07.676802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:07.676802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:07.676802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:07.676802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:07.676802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:07.676802 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:07.677108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:07.677108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:07.677108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.677108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:07.677814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.677814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:07.677814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:07.677814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:07.677814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:07.677814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:07.677814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.677814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:07.677814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:07.677814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.677814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:07.677814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:07.677814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.677814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:07.677814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:07.677814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.677814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:07.678488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:07.678488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:07.678488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:07.678488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.678488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:07.678488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:07.678488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:07.678488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.678488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:07.678488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:07.678488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:07.678488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:07.678488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:07.678488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:07.678488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:07.678488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:07.678488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:07.678488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:07.678488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:07.678978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:07.678978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:07.678978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.678978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:07.682345 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:07.682598 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:07.682755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:07.682946 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:07.683210 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:07.683387 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:07.683505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.683929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:07.683929 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:07.683929 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:07.683929 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:07.683929 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:07.683929 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:07.683929 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.683929 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:07.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:07.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:07.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:07.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:07.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:07.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:07.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:07.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:07.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.685170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:07.685170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:07.685170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:07.685170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:07.685170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:07.685170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:07.685170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:07.685170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:07.685170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:07.685170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:07.685170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:07.685170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:07.685170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:07.685398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.685398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:07.686068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.686068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:07.686068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:07.686068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:07.686068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:07.686068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:07.686068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.686068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:07.686068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:07.686068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.686068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:07.686068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:07.686068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.686068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:07.686068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:07.686068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:07.686731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:07.686731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:07.686731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.686731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:07.686731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:07.686731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:07.686731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:07.686731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:07.686731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:07.686731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:07.686731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:07.686731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:07.686731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:07.686731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:07.686731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:07.686731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:07.686731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:07.686731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:07.686731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:07.687005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:07.687005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:07.687005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:08.862635 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:08.863230 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:08.863230 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:08.863723 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:08.891879 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:08.892103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:08.892659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:08.892659 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:08.892659 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:08.892659 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:08.892659 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:08.892659 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:08.892659 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:08.892659 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:08.893424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:08.893424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:08.893424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:08.893424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:08.893424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:08.893424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:08.893424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:08.893424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:08.893424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:08.893424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:08.893424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:08.893424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:08.893770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:08.893770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:08.893770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:08.893770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:08.893770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:08.893770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:08.893770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:08.893770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:08.893770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:08.893770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:08.893770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:08.893770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:08.893770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:08.894042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:08.894042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:08.894354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:08.894354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:08.894354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:08.894354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:08.894354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:08.894354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:08.894354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:08.894354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:08.894354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:08.894354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:08.894354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:08.894354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:08.894354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:08.894354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:08.894354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:08.894354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:08.895076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:08.895076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:08.895076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:08.895076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:08.895076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:08.895076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:08.895076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:08.895076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:08.895076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:08.895076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:08.895076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:08.895076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:08.895076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:08.895076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:08.895076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:08.895076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:08.895076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:08.895076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:08.895076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:08.895967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:08.895967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:08.895967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:12.660851 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:12.661297 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:12.661297 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:12.661750 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:12.661924 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:12.662042 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:12.662142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:12.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:12.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:12.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:12.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:12.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:12.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.662568 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:12.663208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.663208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:12.663208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:12.663208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:12.663208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.663208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:12.663208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:12.663208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.663208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:12.663208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:12.663208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:12.663208 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.663566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:12.663566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:12.663566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:12.663566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:12.663566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:12.663566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:12.663566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:12.663566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:12.663566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:12.663566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:12.663566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:12.663566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:12.663566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:12.663879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.663879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:12.664147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.664147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:12.664147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:12.664147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:12.664147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:12.664147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:12.664147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.664147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:12.664147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:12.664147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.664147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:12.664147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:12.664147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.664147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:12.664147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:12.664147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.664605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:12.664605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:12.664605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.664605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:12.664605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:12.664605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:12.664605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.664605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:12.664605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:12.664605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:12.664605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:12.664605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:12.664605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:12.664605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:12.664605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:12.664605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:12.664605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:12.664605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:12.664605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:12.665071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:12.665071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.665071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:12.665876 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:12.666006 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:12.666107 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:12.666194 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:49:12.666436 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:12.666568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.666927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:12.666927 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:12.666927 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:12.666927 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:12.666927 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:12.666927 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:12.666927 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.666927 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:12.667420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.667420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:12.667420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:12.667420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:12.667420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.667420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:12.667420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:12.667420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:12.667420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.667420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:12.667420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:12.667420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:12.667420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:12.667420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:12.667420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:12.667869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:12.667869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:12.667869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:12.667869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:12.667869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:12.667869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:12.667869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:12.667869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.667869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:12.668112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.668112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:12.668112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:12.668112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:12.668112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.668112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:12.668112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:12.668112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:49:12.668112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:12.668112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:12.668112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.668112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:12.668466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:12.668466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:12.668466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.668466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:12.668466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:12.668466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:12.668466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:12.668466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:12.668466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:12.668466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:12.668466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:12.668466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:12.668466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:12.668466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:12.668867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:12.668867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:12.668867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.668867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:12.668867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.668867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:12.668867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:12.668867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:12.668867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:12.668867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:12.668867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.668867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:12.668867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:12.668867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.669228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:12.669228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:12.669228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.669228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:12.669228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:12.669228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.669228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:12.669228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:12.669228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:49:12.669228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:12.669228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:12.669228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.669228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:12.669228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:12.669228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:12.669228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.669228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:12.669228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:12.669228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:12.669759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:12.669759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:12.669759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:12.669759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:12.669759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:12.669759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:12.669759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:12.669759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:12.669759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:12.669759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:12.669759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.669759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:12.673474 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:12.673623 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:12.673715 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:12.673814 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:12.674080 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:12.674267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.674678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:12.674678 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:12.674678 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:12.674678 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:12.674678 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:12.674678 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:12.674678 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.674678 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:12.675163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.675163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:12.675163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:12.675163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:12.675163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.675163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:12.675163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:12.675163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:12.675163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.675163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:12.675163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:12.675163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:12.675163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:12.675163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:12.675163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:12.675585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:12.675585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:12.675585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:12.675585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:12.675585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:12.675585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:12.675585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:12.675585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.675585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:12.675814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.675814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:12.675814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:12.675814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:12.675814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.675814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:12.675814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:12.675814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:12.675814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:12.675814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.675814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:12.676115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:12.676115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:12.676115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.676115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:12.676115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:12.676115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:12.676115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:12.676115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:12.676115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:12.676115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:12.676115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:12.676115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:12.676115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:12.676115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:12.676480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:12.676480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:12.676480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.676480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:12.676730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.676730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:12.676730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:12.676730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:12.676730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:12.676730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:12.676730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.676730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:12.676730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:12.676730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.676730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:12.676730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:12.676730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.676730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:12.676730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:12.676730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.676730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:12.677152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:12.677152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:12.677152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:12.677152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.677152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:12.677152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:12.677152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:12.677152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.677152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:12.677152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:12.677152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:12.677152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:12.677152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:12.677152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:12.677152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:12.677152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:12.677152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:12.677152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:12.677152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:12.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:12.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:12.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:12.682734 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:12.682900 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:12.683002 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:12.683129 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:12.683393 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:12.683570 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:12.683707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.684071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:12.684071 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:12.684071 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:12.684071 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:12.684071 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:12.684071 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:12.684071 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.684071 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:12.684623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.684623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:12.684623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:12.684623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:12.684623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.684623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:12.684623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:12.684623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.684623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:12.684623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:12.684623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:12.684623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.684947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:12.684947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:12.684947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:12.684947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:12.684947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:12.684947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:12.684947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:12.684947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:12.684947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:12.684947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:12.684947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:12.684947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:12.684947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:12.685244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.685244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:12.685244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.685244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:12.685244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:12.685244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:12.685244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:12.685244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:12.685244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.685244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:12.685244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:12.685244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.685244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:12.685244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:12.685244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.685671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:12.685671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:12.685671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:12.685671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:12.685671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:12.685671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.685671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:12.685671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:12.685671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:12.685671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:12.685671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:12.685671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:12.685671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:12.685671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:12.685671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:12.685671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:12.685671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:12.685671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:12.685671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:12.686131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:12.686131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:12.686131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:12.686131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:12.686131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:12.686131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:13.863026 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:13.863492 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:13.863528 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:13.864201 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:13.892147 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:13.892350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:13.892911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:13.892911 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:13.892911 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:13.892911 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:13.892911 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:13.892911 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:13.892911 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:13.892911 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:13.893612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:13.893612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:13.893612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:13.893612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:13.893612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:13.893612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:13.893612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:13.893612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:13.893612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:13.893612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:13.893612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:13.893612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:13.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:13.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:13.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:13.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:13.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:13.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:13.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:13.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:13.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:13.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:13.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:13.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:13.893846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:13.894053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:13.894053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:13.894381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:13.894381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:13.894381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:13.894381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:13.894381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:13.894381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:13.894381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:13.894381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:13.894381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:13.894381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:13.894381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:13.894381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:13.894381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:13.894381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:13.894381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:13.894381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:13.894736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:13.894736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:13.894736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:13.894736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:13.894736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:13.894736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:13.894736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:13.894736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:13.894736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:13.894736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:13.894736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:13.894736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:13.894736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:13.894736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:13.894736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:13.894736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:13.894736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:13.894736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:13.894736 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:13.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:13.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:13.895051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:17.661291 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:17.661830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:17.661830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:17.662462 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:17.662599 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:17.662751 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:17.662842 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.663327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:17.663327 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:17.663327 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:17.663327 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:17.663327 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:17.663327 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:17.663327 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.663327 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:17.664311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.664311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:17.664311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:17.664311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:17.664311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.664311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:17.664311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:17.664311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.664311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:17.664311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:17.664311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:17.664311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.664815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:17.664815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:17.664815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:17.664815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:17.664815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:17.664815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:17.664815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:17.664815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:17.664815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:17.664815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:17.664815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:17.664815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:17.664815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:17.665199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.665199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:17.665387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.665387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:17.665387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:17.665387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:17.665387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:17.665387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:17.665387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.665387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:17.665387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:17.665387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.665387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:17.665387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:17.665387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.665387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:17.665387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:17.665387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.665666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:17.665666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:17.665666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.665666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:17.665666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:17.665666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:17.665666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.665666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:17.665666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:17.665666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:17.665666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:17.665666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:17.665666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:17.665666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:17.665666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:17.665666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:17.665666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:17.665666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:17.665666 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:17.666009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:17.666009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.666009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:17.666292 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:17.666524 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:17.666747 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:17.666851 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:49:17.667166 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:17.667349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.667717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:17.667717 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:17.667717 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:17.667717 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:17.667717 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:17.667717 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:17.667717 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.667717 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:17.668457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.668457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:17.668457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:17.668457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:17.668457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.668457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:17.668457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:17.668457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:17.668457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.668457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:17.668457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:17.668457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:17.668457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:17.668457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:17.668457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:17.668951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:17.668951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:17.668951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:17.668951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:17.668951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:17.668951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:17.668951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:17.668951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.668951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:17.669262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.669262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:17.669262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:17.669262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:17.669262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.669262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:17.669262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:17.669262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:49:17.669262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:17.669262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:17.669262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.669262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:17.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:17.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:17.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:17.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:17.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:17.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:17.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:17.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:17.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:17.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:17.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:17.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:17.669627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:17.669840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:17.669840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:17.669840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.669840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:17.670282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.670282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:17.670282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:17.670282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:17.670282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:17.670282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:17.670282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.670282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:17.670282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:17.670282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.670282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:17.670282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:17.670282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.670282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:17.670282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:17.670282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.670563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:17.670563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:17.670563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:49:17.670563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:17.670563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:17.670563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.670563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:17.670563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:17.670563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:17.670563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.670563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:17.670563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:17.670563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:17.670563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:17.670563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:17.670563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:17.670563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:17.670563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:17.670846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:17.670846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:17.670846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:17.670846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:17.670846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:17.670846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.670846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:17.673830 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:17.674083 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:17.674194 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:17.674337 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:17.674632 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:17.674837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.675225 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:17.675225 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:17.675225 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:17.675225 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:17.675225 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:17.675225 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:17.675225 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.675225 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:17.675810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.675810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:17.675810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:17.675810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:17.675810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.675810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:17.675810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:17.675810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:17.675810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.675810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:17.675810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:17.675810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:17.675810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:17.675810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:17.675810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:17.676191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:17.676191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:17.676191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:17.676191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:17.676191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:17.676191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:17.676191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:17.676191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.676191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:17.676452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.676452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:17.676452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:17.676452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:17.676452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.676452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:17.676452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:17.676452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:17.676452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:17.676452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.676452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:17.676753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:17.676753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:17.676753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.676753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:17.676753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:17.676753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:17.676753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:17.676753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:17.676753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:17.676753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:17.676753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:17.676753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:17.676753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:17.676753 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:17.677125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:17.677125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:17.677125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.677125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:17.677222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.677222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:17.677222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:17.677222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:17.677222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:17.677222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:17.677222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.677222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:17.677222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:17.677222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.677222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:17.677222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:17.677222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.677222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:17.677222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:17.677222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.677222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:17.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:17.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:17.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:17.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:17.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:17.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:17.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:17.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:17.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:17.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:17.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:17.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:17.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:17.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:17.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:17.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:17.677631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:17.678093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:17.678093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:17.678093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.678093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:17.683049 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:17.683378 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:17.683512 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:17.683684 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:17.684013 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:17.684170 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:17.684311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.684778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:17.684778 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:17.684778 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:17.684778 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:17.684778 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:17.684778 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:17.684778 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.684778 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:17.685397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.685397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:17.685397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:17.685397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:17.685397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.685397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:17.685397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:17.685397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.685397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:17.685397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:17.685397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:17.685397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.685712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:17.685712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:17.685712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:17.685712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:17.685712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:17.685712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:17.685712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:17.685712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:17.685712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:17.685712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:17.685712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:17.685712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:17.685712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:17.686055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.686055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:17.686055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.686055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:17.686055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:17.686055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:17.686055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:17.686055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:17.686055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.686055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:17.686055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:17.686055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.686055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:17.686055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:17.686055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:17.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:17.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:17.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:17.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:17.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:17.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:17.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:17.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:17.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:17.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:17.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:17.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:17.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:17.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:17.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:17.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:17.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:17.686939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:17.686939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:17.686939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:17.686939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:17.686939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:17.686939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:18.863566 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:18.864317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:18.864317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:18.864757 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:18.895276 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:18.895509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:18.895959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:18.895959 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:18.895959 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:18.895959 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:18.895959 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:18.895959 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:18.895959 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:18.895959 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:18.896866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:18.896866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:18.896866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:18.896866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:18.896866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:18.896866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:18.896866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:18.896866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:18.896866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:18.896866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:18.896866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:18.896866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:18.897308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:18.897308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:18.897308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:18.897308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:18.897308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:18.897308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:18.897308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:18.897308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:18.897308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:18.897308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:18.897308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:18.897308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:18.897308 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:18.897551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:18.897551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:18.897955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:18.897955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:18.897955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:18.897955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:18.897955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:18.897955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:18.897955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:18.897955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:18.897955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:18.897955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:18.897955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:18.897955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:18.897955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:18.897955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:18.897955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:18.897955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:18.898849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:18.898849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:18.898849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:18.898849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:18.898849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:18.898849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:18.898849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:18.898849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:18.898849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:18.898849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:18.898849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:18.898849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:18.898849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:18.898849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:18.898849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:18.898849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:18.898849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:18.898849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:18.898849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:18.899442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:18.899442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:18.899442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:22.661726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:22.662136 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:22.662136 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:22.662562 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:22.662737 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:22.662847 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:22.662950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.663352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:22.663352 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:22.663352 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:22.663352 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:22.663352 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:22.663352 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:22.663352 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.663352 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:22.664046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.664046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:22.664046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:22.664046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:22.664046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.664046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:22.664046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:22.664046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.664046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:22.664046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:22.664046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:22.664046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:22.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:22.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:22.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:22.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:22.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:22.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:22.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:22.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:22.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:22.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:22.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:22.664297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:22.664507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.664507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:22.664881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.664881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:22.664881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:22.664881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:22.664881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:22.664881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:22.664881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.664881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:22.664881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:22.664881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.664881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:22.664881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:22.664881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.664881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:22.664881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:22.664881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.665157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:22.665157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:22.665157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.665157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:22.665157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:22.665157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:22.665157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.665157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:22.665157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:22.665157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:22.665157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:22.665157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:22.665157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:22.665157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:22.665157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:22.665157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:22.665157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:22.665157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:22.665157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:22.665455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:22.665455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.665455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:22.666515 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:22.666676 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:22.666760 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:22.666841 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:49:22.667082 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:22.667215 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.667554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:22.667554 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:22.667554 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:22.667554 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:22.667554 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:22.667554 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:22.667554 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.667554 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:22.668087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.668087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:22.668087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:22.668087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:22.668087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.668087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:22.668087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:22.668087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:22.668087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.668087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:22.668087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:22.668087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:22.668087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:22.668087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:22.668087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:22.668364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:22.668364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:22.668364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:22.668364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:22.668364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:22.668364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:22.668364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:22.668364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.668364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:22.668739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.668739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:22.668739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:22.668739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:22.668739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.668739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:22.668739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:22.668739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:49:22.668739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:22.668739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:22.668739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.668739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:22.669150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:22.669150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:22.669150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.669150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:22.669150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:22.669150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:22.669150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:22.669150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:22.669150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:22.669150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:22.669150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:22.669150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:22.669150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:22.669150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:22.669367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:22.669367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:22.669367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.669367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:22.669367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.669367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:22.669367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:22.669367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:22.669367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:22.669367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:22.669367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.669367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:22.669367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:22.669367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.669584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:22.669584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:22.669584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.669584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:22.669584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:22.669584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.669584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:22.669584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:22.669584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:49:22.669584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:22.669584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:22.669584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.669584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:22.669584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:22.669584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:22.669584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.669584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:22.669584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:22.669584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:22.669887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:22.669887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:22.669887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:22.669887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:22.669887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:22.669887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:22.669887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:22.669887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:22.669887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:22.669887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:22.669887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.669887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:22.674067 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:22.674211 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:22.674311 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:22.674391 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:22.674610 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:22.674785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.675084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:22.675084 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:22.675084 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:22.675084 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:22.675084 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:22.675084 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:22.675084 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.675084 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:22.675580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.675580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:22.675580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:22.675580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:22.675580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.675580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:22.675580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:22.675580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:22.675580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.675580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:22.675580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:22.675580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:22.675580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:22.675580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:22.675580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:22.675859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:22.675859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:22.675859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:22.675859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:22.675859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:22.675859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:22.675859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:22.675859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.675859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:22.676212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.676212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:22.676212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:22.676212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:22.676212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.676212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:22.676212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:22.676212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:22.676212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:22.676212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.676212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:22.676422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:22.676422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:22.676422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.676422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:22.676422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:22.676422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:22.676422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:22.676422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:22.676422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:22.676422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:22.676422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:22.676422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:22.676422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:22.676422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:22.676617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:22.676617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:22.676617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.676617 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:22.676956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.676956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:22.676956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:22.676956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:22.676956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:22.676956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:22.676956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.676956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:22.676956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:22.676956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.676956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:22.676956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:22.676956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.676956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:22.676956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:22.676956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.676956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:22.677228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:22.677228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:22.677228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:22.677228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.677228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:22.677228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:22.677228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:22.677228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.677228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:22.677228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:22.677228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:22.677228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:22.677228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:22.677228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:22.677228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:22.677228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:22.677228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:22.677228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:22.677228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:22.677531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:22.677531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:22.677531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.677531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:22.683327 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:22.683487 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:22.683566 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:22.683696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:22.683940 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:22.684091 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:22.684206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.684595 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:22.684595 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:22.684595 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:22.684595 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:22.684595 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:22.684595 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:22.684595 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.684595 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:22.685132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.685132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:22.685132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:22.685132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:22.685132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.685132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:22.685132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:22.685132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.685132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:22.685132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:22.685132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:22.685132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.685453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:22.685453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:22.685453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:22.685453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:22.685453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:22.685453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:22.685453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:22.685453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:22.685453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:22.685453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:22.685453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:22.685453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:22.685453 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:22.685747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.685747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:22.685803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.685803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:22.685803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:22.685803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:22.685803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:22.685803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:22.685803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.685803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:22.685803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:22.685803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.685803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:22.685803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:22.685803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.685803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:22.685803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:22.685803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:22.686163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:22.686163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:22.686163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.686163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:22.686163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:22.686163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:22.686163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:22.686163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:22.686163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:22.686163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:22.686163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:22.686163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:22.686163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:22.686163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:22.686163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:22.686163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:22.686163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:22.686163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:22.686163 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:22.686594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:22.686594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:22.686594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:23.864112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:23.864584 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:23.864584 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:23.865583 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:23.897911 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:23.898155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:23.898599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:23.898599 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:23.898599 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:23.898599 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:23.898599 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:23.898599 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:23.898599 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:23.898599 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:23.899245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:23.899245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:23.899245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:23.899245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:23.899245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:23.899245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:23.899245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:23.899245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:23.899245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:23.899245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:23.899245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:23.899245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:23.899624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:23.899624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:23.899624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:23.899624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:23.899624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:23.899624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:23.899624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:23.899624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:23.899624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:23.899624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:23.899624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:23.899624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:23.899624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:23.899937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:23.899937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:23.900001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:23.900001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:23.900001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:23.900001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:23.900001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:23.900001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:23.900001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:23.900001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:23.900001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:23.900001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:23.900001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:23.900001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:23.900001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:23.900001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:23.900001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:23.900001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:23.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:23.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:23.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:23.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:23.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:23.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:23.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:23.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:23.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:23.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:23.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:23.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:23.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:23.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:23.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:23.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:23.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:23.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:23.900404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:23.900863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:23.900863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:23.900863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:27.662091 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:27.662767 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:27.662767 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:27.663338 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:27.663507 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:27.663692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:27.663865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.664455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:27.664455 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:27.664455 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:27.664455 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:27.664455 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:27.664455 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:27.664455 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.664455 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:27.665274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.665274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:27.665274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:27.665274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:27.665274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.665274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:27.665274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:27.665274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.665274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:27.665274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:27.665274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:27.665274 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.665638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:27.665638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:27.665638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:27.665638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:27.665638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:27.665638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:27.665638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:27.665638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:27.665638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:27.665638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:27.665638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:27.665638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:27.665638 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:27.666019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.666019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:27.666467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.666467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:27.666467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:27.666467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:27.666467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:27.666467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:27.666467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.666467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:27.666467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:27.666467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.666467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:27.666467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:27.666467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.666467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:27.666467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:27.666467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.666920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:27.666920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:27.666920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.666920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:27.666920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:27.666920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:27.666920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.666920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:27.666920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:27.666920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:27.666920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:27.666920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:27.666920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:27.666920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:27.666920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:27.666920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:27.666920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:27.666920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:27.666920 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:27.667438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:27.667438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.667438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:27.667438 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:27.667438 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:27.667438 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:27.667438 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:49:27.667564 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:27.667955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.668320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:27.668320 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:27.668320 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:27.668320 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:27.668320 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:27.668320 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:27.668320 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.668320 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:27.669160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.669160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:27.669160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:27.669160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:27.669160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.669160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:27.669160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:27.669160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:27.669160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.669160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:27.669160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:27.669160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:27.669160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:27.669160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:27.669160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:27.670030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:27.670030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:27.670030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:27.670030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:27.670030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:27.670030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:27.670030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:27.670030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.670030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:27.670473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.670473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:27.670473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:27.670473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:27.670473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.670473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:27.670473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:27.670473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:49:27.670473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:27.670473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:27.670473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.670473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:27.670957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:27.670957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:27.670957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.670957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:27.670957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:27.670957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:27.670957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:27.670957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:27.670957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:27.670957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:27.670957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:27.670957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:27.670957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:27.670957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:27.671161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:27.671161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:27.671161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.671161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:27.671427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.671427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:27.671427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:27.671427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:27.671427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:27.671427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:27.671427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.671427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:27.671427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:27.671427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.671427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:27.671427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:27.671427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.671427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:27.671427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:27.671427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.671875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:27.671875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:27.671875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:49:27.671875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:27.671875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:27.671875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.671875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:27.671875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:27.671875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:27.671875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.671875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:27.671875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:27.671875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:27.671875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:27.671875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:27.671875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:27.671875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:27.671875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:27.672147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:27.672147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:27.672147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:27.672147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:27.672147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:27.672147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.672147 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:27.674365 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:27.674604 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:27.674738 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:27.674884 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:27.675244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:27.675517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.676038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:27.676038 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:27.676038 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:27.676038 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:27.676038 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:27.676038 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:27.676038 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.676038 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:27.676839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.676839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:27.676839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:27.676839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:27.676839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.676839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:27.676839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:27.676839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:27.676839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.676839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:27.676839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:27.676839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:27.676839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:27.676839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:27.676839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:27.677103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:27.677103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:27.677103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:27.677103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:27.677103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:27.677103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:27.677103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:27.677103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.677103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:27.677838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.677838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:27.677838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:27.677838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:27.677838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.677838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:27.677838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:27.677838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:27.677838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:27.677838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.677838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:27.678046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:27.678046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:27.678046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.678046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:27.678046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:27.678046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:27.678046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:27.678046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:27.678046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:27.678046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:27.678046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:27.678046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:27.678046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:27.678046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:27.678330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:27.678330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:27.678330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.678330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:27.679077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.679077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:27.679077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:27.679077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:27.679077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:27.679077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:27.679077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.679077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:27.679077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:27.679077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.679077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:27.679077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:27.679077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.679077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:27.679077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:27.679077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.679077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:27.679711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:27.679711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:27.679711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:27.679711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.679711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:27.679711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:27.679711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:27.679711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.679711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:27.679711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:27.679711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:27.679711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:27.679711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:27.679711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:27.679711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:27.679711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:27.679711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:27.679711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:27.679711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:27.680125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:27.680125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:27.680125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.680125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:27.683756 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:27.684090 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:27.684228 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:27.684418 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:27.684748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:27.684950 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:27.685066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.685537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:27.685537 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:27.685537 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:27.685537 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:27.685537 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:27.685537 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:27.685537 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.685537 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:27.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:27.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:27.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:27.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:27.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:27.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:27.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:27.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:27.686442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.686790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:27.686790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:27.686790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:27.686790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:27.686790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:27.686790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:27.686790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:27.686790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:27.686790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:27.686790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:27.686790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:27.686790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:27.686790 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:27.687078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.687078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:27.687527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.687527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:27.687527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:27.687527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:27.687527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:27.687527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:27.687527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.687527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:27.687527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:27.687527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.687527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:27.687527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:27.687527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.687527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:27.687527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:27.687527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:27.687942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:27.687942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:27.687942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.687942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:27.687942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:27.687942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:27.687942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:27.687942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:27.687942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:27.687942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:27.687942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:27.687942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:27.687942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:27.687942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:27.687942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:27.687942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:27.687942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:27.687942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:27.687942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:27.688367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:27.688367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:27.688367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:28.864545 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:28.865034 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:28.865034 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:28.865405 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:28.893737 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:28.893961 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:28.894522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:28.894522 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:28.894522 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:28.894522 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:28.894522 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:28.894522 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:28.894522 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:28.894522 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:28.895223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:28.895223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:28.895223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:28.895223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:28.895223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:28.895223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:28.895223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:28.895223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:28.895223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:28.895223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:28.895223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:28.895223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:28.895543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:28.895543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:28.895543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:28.895543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:28.895543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:28.895543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:28.895543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:28.895543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:28.895543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:28.895543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:28.895543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:28.895543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:28.895543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:28.895779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:28.895779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:28.896157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:28.896157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:28.896157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:28.896157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:28.896157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:28.896157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:28.896157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:28.896157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:28.896157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:28.896157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:28.896157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:28.896157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:28.896157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:28.896157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:28.896157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:28.896157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:28.896463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:28.896463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:28.896463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:28.896463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:28.896463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:28.896463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:28.896463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:28.896463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:28.896463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:28.896463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:28.896463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:28.896463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:28.896463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:28.896463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:28.896463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:28.896463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:28.896463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:28.896463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:28.896463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:28.896838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:28.896838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:28.896838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:32.662522 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:32.663106 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:32.663106 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:32.663648 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:32.663818 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:32.663981 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:32.664110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.664677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:32.664677 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:32.664677 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:32.664677 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:32.664677 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:32.664677 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:32.664677 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.664677 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:32.665352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.665352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:32.665352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:32.665352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:32.665352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.665352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:32.665352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:32.665352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.665352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:32.665352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:32.665352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:32.665352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.665722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:32.665722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:32.665722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:32.665722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:32.665722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:32.665722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:32.665722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:32.665722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:32.665722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:32.665722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:32.665722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:32.665722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:32.665722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:32.666058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.666058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:32.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:32.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:32.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:32.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:32.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:32.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:32.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:32.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:32.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:32.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:32.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:32.666128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.666589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:32.666589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:32.666589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.666589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:32.666589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:32.666589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:32.666589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.666589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:32.666589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:32.666589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:32.666589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:32.666589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:32.666589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:32.666589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:32.666589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:32.666589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:32.666589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:32.666589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:32.666589 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:32.667100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:32.667100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.667100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:32.667100 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:32.667354 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:32.667468 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:32.667602 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:49:32.667939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:32.668119 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.668558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:32.668558 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:32.668558 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:32.668558 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:32.668558 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:32.668558 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:32.668558 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.668558 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:32.669373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.669373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:32.669373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:32.669373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:32.669373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.669373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:32.669373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:32.669373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:32.669373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.669373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:32.669373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:32.669373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:32.669373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:32.669373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:32.669373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:32.669817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:32.669817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:32.669817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:32.669817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:32.669817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:32.669817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:32.669817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:32.669817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.669817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:32.670297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.670297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:32.670297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:32.670297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:32.670297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.670297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:32.670297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:32.670297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:49:32.670297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:32.670297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:32.670297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.670297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:32.670667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:32.670667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:32.670667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.670667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:32.670667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:32.670667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:32.670667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:32.670667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:32.670667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:32.670667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:32.670667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:32.670667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:32.670667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:32.670667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:32.671022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:32.671022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:32.671022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.671022 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:32.671406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.671406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:32.671406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:32.671406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:32.671406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:32.671406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:32.671406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.671406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:32.671406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:32.671406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.671406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:32.671406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:32.671406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.671406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:32.671406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:32.671406 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.671873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:32.671873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:32.671873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:49:32.671873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:32.671873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:32.671873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.671873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:32.671873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:32.671873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:32.671873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.671873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:32.671873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:32.671873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:32.671873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:32.671873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:32.671873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:32.671873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:32.671873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:32.672314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:32.672314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:32.672314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:32.672314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:32.672314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:32.672314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.672314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:32.674700 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:32.674953 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:32.675093 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:32.675262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:32.675570 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:32.675789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.676212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:32.676212 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:32.676212 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:32.676212 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:32.676212 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:32.676212 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:32.676212 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.676212 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:32.677085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.677085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:32.677085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:32.677085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:32.677085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.677085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:32.677085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:32.677085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:32.677085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.677085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:32.677085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:32.677085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:32.677085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:32.677085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:32.677085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:32.677506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:32.677506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:32.677506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:32.677506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:32.677506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:32.677506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:32.677506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:32.677506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.677506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:32.678084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.678084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:32.678084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:32.678084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:32.678084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.678084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:32.678084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:32.678084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:32.678084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:32.678084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.678084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:32.678426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:32.678426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:32.678426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.678426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:32.678426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:32.678426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:32.678426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:32.678426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:32.678426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:32.678426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:32.678426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:32.678426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:32.678426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:32.678426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:32.678817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:32.678817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:32.678817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.678817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:32.679342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.679342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:32.679342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:32.679342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:32.679342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:32.679342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:32.679342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.679342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:32.679342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:32.679342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.679342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:32.679342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:32.679342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.679342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:32.679342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:32.679342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.679342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:32.679772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:32.679772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:32.679772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:32.679772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.679772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:32.679772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:32.679772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:32.679772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.679772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:32.679772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:32.679772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:32.679772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:32.679772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:32.679772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:32.679772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:32.679772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:32.679772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:32.679772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:32.679772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:32.680269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:32.680269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:32.680269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.680269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:32.684116 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:32.684398 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:32.684539 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:32.684745 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:32.685023 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:32.685229 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:32.685409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.685849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:32.685849 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:32.685849 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:32.685849 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:32.685849 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:32.685849 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:32.685849 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.685849 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:32.686758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.686758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:32.686758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:32.686758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:32.686758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.686758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:32.686758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:32.686758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.686758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:32.686758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:32.686758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:32.686758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.687051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:32.687051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:32.687051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:32.687051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:32.687051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:32.687051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:32.687051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:32.687051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:32.687051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:32.687051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:32.687051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:32.687051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:32.687051 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:32.687383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.687383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:32.687922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.687922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:32.687922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:32.687922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:32.687922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:32.687922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:32.687922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.687922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:32.687922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:32.687922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.687922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:32.687922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:32.687922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.687922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:32.687922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:32.687922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:32.688385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:32.688385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:32.688385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.688385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:32.688385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:32.688385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:32.688385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:32.688385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:32.688385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:32.688385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:32.688385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:32.688385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:32.688385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:32.688385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:32.688385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:32.688385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:32.688385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:32.688385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:32.688385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:32.688941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:32.688941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:32.688941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:33.864883 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:33.865365 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:33.865365 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:33.865840 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:33.898270 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:33.898446 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:33.899303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:33.899303 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:33.899303 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:33.899303 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:33.899303 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:33.899303 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:33.899303 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:33.899303 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:33.899901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:33.899901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:33.899901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:33.899901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:33.899901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:33.899901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:33.899901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:33.899901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:33.899901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:33.899901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:33.899901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:33.899901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:33.900230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:33.900230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:33.900230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:33.900230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:33.900230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:33.900230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:33.900230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:33.900230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:33.900230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:33.900230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:33.900230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:33.900230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:33.900230 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:33.900588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:33.900588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:33.901075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:33.901075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:33.901075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:33.901075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:33.901075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:33.901075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:33.901075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:33.901075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:33.901075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:33.901075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:33.901075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:33.901075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:33.901075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:33.901075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:33.901075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:33.901075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:33.901503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:33.901503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:33.901503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:33.901503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:33.901503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:33.901503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:33.901503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:33.901503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:33.901503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:33.901503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:33.901503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:33.901503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:33.901503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:33.901503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:33.901503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:33.901503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:33.901503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:33.901503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:33.901503 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:33.901993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:33.901993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:33.901993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:37.662961 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:37.663738 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:37.663738 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:37.663971 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:37.664191 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:37.664401 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:37.664686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.665237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:37.665237 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:37.665237 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:37.665237 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:37.665237 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:37.665237 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:37.665237 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.665237 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:37.666176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.666176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:37.666176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:37.666176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:37.666176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.666176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:37.666176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:37.666176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.666176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:37.666176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:37.666176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:37.666176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.666561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:37.666561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:37.666561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:37.666561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:37.666561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:37.666561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:37.666561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:37.666561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:37.666561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:37.666561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:37.666561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:37.666561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:37.666561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:37.667006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.667006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:37.667006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.667006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:37.667006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:37.667006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:37.667006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:37.667006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:37.667006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.667006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:37.667006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:37.667006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.667006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:37.667006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:37.667006 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.667436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:37.667436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:37.667436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.667436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:37.667436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:37.667436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.667436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:37.667436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:37.667436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:37.667436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.667436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:37.667436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:37.667436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:37.667436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:37.667436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:37.667436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:37.667436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:37.667436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:37.667436 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:37.668008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:37.668008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:37.668008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:37.668008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:37.668008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.668008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:37.668008 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:37.668008 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:37.668008 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:37.668008 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:49:37.668307 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:37.668307 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.668735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:37.668735 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:37.668735 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:37.668735 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:37.668735 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:37.668735 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:37.668735 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.668735 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:37.669409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.669409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:37.669409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:37.669409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:37.669409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.669409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:37.669409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:37.669409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:37.669409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.669409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:37.669409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:37.669409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:37.669409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:37.669409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:37.669409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:37.669837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:37.669837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:37.669837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:37.669837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:37.669837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:37.669837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:37.669837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:37.669837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.669837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:37.670132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.670132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:37.670132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:37.670132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:37.670132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.670132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:37.670132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:37.670132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:49:37.670132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:37.670132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:37.670132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.670132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:37.670506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:37.670506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:37.670506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.670506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:37.670506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:37.670506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:37.670506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:37.670506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:37.670506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:37.670506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:37.670506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:37.670506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:37.670506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:37.670506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:37.670836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:37.670836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:37.670836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.670836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:37.671020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.671020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:37.671020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:37.671020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:37.671020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:37.671020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:37.671020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.671020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:37.671020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:37.671020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.671020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:37.671020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:37.671020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.671020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:37.671020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:37.671020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.671400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:37.671400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:37.671400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:49:37.671400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:37.671400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:37.671400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.671400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:37.671400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:37.671400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:37.671400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.671400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:37.671400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:37.671400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:37.671400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:37.671400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:37.671400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:37.671400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:37.671400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:37.671793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:37.671793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:37.671793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:37.671793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:37.671793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:37.671793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.671793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:37.675065 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:37.675268 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:37.675393 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:37.675509 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:37.675890 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:37.676131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.676695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:37.676695 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:37.676695 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:37.676695 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:37.676695 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:37.676695 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:37.676695 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.676695 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:37.677447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.677447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:37.677447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:37.677447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:37.677447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.677447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:37.677447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:37.677447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:37.677447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.677447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:37.677447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:37.677447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:37.677447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:37.677447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:37.677447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:37.677893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:37.677893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:37.677893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:37.677893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:37.677893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:37.677893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:37.677893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:37.677893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.677893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:37.678403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.678403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:37.678403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:37.678403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:37.678403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.678403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:37.678403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:37.678403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:37.678403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:37.678403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.678403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:37.678781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:37.678781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:37.678781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.678781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:37.678781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:37.678781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:37.678781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:37.678781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:37.678781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:37.678781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:37.678781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:37.678781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:37.678781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:37.678781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:37.679168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:37.679168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:37.679168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.679168 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:37.679521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.679521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:37.679521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:37.679521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:37.679521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:37.679521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:37.679521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.679521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:37.679521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:37.679521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.679521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:37.679521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:37.679521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.679521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:37.679521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:37.679521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.679521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:37.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:37.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:37.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:37.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:37.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:37.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:37.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:37.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:37.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:37.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:37.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:37.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:37.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:37.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:37.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:37.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:37.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:37.680712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:37.680712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:37.680712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.680712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:37.684452 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:37.684686 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:37.684812 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:37.684978 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:37.685303 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:37.685530 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:37.685727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.686183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:37.686183 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:37.686183 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:37.686183 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:37.686183 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:37.686183 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:37.686183 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.686183 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:37.686891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.686891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:37.686891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:37.686891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:37.686891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.686891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:37.686891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:37.686891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.686891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:37.686891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:37.686891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:37.686891 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.687301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:37.687301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:37.687301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:37.687301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:37.687301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:37.687301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:37.687301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:37.687301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:37.687301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:37.687301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:37.687301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:37.687301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:37.687301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:37.687636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.687636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:37.687955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.687955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:37.687955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:37.687955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:37.687955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:37.687955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:37.687955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.687955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:37.687955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:37.687955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.687955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:37.687955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:37.687955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.687955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:37.687955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:37.687955 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:37.688288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:37.688288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:37.688288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.688288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:37.688288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:37.688288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:37.688288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:37.688288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:37.688288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:37.688288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:37.688288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:37.688288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:37.688288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:37.688288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:37.688288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:37.688288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:37.688288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:37.688288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:37.688288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:37.688672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:37.688672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:37.688672 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:38.865380 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:38.865980 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:38.865980 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:38.866765 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:38.903420 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:38.903661 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:38.904334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:38.904334 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:38.904334 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:38.904334 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:38.904334 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:38.904334 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:38.904334 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:38.904334 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:38.905424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:38.905424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:38.905424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:38.905424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:38.905424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:38.905424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:38.905424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:38.905424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:38.905424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:38.905424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:38.905424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:38.905424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:38.905877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:38.905877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:38.905877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:38.905877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:38.905877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:38.905877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:38.905877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:38.905877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:38.905877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:38.905877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:38.905877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:38.905877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:38.905877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:38.906280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:38.906280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:38.906280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:38.906280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:38.906280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:38.906280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:38.906280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:38.906280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:38.906280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:38.906280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:38.906280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:38.906280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:38.906280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:38.906280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:38.906280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:38.906755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:38.906755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:38.906755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:38.906755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:38.906755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:38.906755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:38.906755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:38.906755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:38.906755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:38.906755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:38.906755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:38.906755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:38.906755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:38.906755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:38.906755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:38.906755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:38.906755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:38.906755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:38.906755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:38.907291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:38.907291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:38.907291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:38.907291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:38.907291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:38.907291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:42.663340 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:42.663702 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:42.663794 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:42.664452 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:42.664708 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:42.664868 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:42.665021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.665624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:42.665624 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:42.665624 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:42.665624 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:42.665624 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:42.665624 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:42.665624 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.665624 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:42.666454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.666454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:42.666454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:42.666454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:42.666454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.666454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:42.666454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:42.666454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.666454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:42.666454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:42.666454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:42.666454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.666682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:42.666682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:42.666682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:42.666682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:42.666682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:42.666682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:42.666682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:42.666682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:42.666682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:42.666682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:42.666682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:42.666682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:42.666682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:42.666902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.666902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:42.667721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.667721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:42.667721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:42.667721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:42.667721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:42.667721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:42.667721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.667721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:42.667721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:42.667721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.667721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:42.667721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:42.667721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.667721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:42.667721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:42.667721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.668015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:42.668015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:42.668015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.668015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:42.668015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:42.668015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:42.668015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.668015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:42.668015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:42.668015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:42.668015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:42.668015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:42.668015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:42.668015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:42.668015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:42.668015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:42.668015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:42.668015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:42.668015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:42.668333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:42.668333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.668333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:42.668333 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:42.668587 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:42.668731 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:42.668856 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:49:42.669190 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:42.669451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.670017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:42.670017 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:42.670017 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:42.670017 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:42.670017 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:42.670017 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:42.670017 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.670017 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:42.670605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.670605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:42.670605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:42.670605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:42.670605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.670605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:42.670605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:42.670605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:42.670605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.670605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:42.670605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:42.670605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:42.670605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:42.670605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:42.670605 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:42.670859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:42.670859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:42.670859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:42.670859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:42.670859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:42.670859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:42.670859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:42.670859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.670859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:42.671236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.671236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:42.671236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:42.671236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:42.671236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.671236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:42.671236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:42.671236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:49:42.671236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:42.671236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:42.671236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.671236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:42.671459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:42.671459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:42.671459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.671459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:42.671459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:42.671459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:42.671459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:42.671459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:42.671459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:42.671459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:42.671459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:42.671459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:42.671459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:42.671459 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:42.671741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:42.671741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:42.671741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.671741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:42.672146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.672146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:42.672146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:42.672146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:42.672146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:42.672146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:42.672146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.672146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:42.672146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:42.672146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.672146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:42.672146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:42.672146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.672146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:42.672146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:42.672146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.672584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:42.672584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:42.672584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:49:42.672584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:42.672584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:42.672584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.672584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:42.672584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:42.672584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:42.672584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.672584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:42.672584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:42.672584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:42.672584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:42.672584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:42.672584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:42.672584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:42.672584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:42.673031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:42.673031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:42.673031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:42.673031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:42.673031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:42.673031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.673031 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:42.675320 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:42.675569 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:42.675677 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:42.675783 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:42.676101 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:42.676407 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.676960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:42.676960 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:42.676960 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:42.676960 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:42.676960 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:42.676960 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:42.676960 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.676960 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:42.677593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.677593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:42.677593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:42.677593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:42.677593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.677593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:42.677593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:42.677593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:42.677593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.677593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:42.677593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:42.677593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:42.677593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:42.677593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:42.677593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:42.677882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:42.677882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:42.677882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:42.677882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:42.677882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:42.677882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:42.677882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:42.677882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.677882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:42.678502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.678502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:42.678502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:42.678502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:42.678502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.678502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:42.678502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:42.678502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:42.678502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:42.678502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.678502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:42.678715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:42.678715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:42.678715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.678715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:42.678715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:42.678715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:42.678715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:42.678715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:42.678715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:42.678715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:42.678715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:42.678715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:42.678715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:42.678715 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:42.679015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:42.679015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:42.679015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.679015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:42.679244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.679244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:42.679244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:42.679244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:42.679244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:42.679244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:42.679244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.679244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:42.679244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:42.679244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.679244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:42.679244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:42.679244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.679244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:42.679244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:42.679244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.679244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:42.679561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:42.679561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:42.679561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:42.679561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.679561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:42.679561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:42.679561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:42.679561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.679561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:42.679561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:42.679561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:42.679561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:42.679561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:42.679561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:42.679561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:42.679561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:42.679561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:42.679561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:42.679561 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:42.679865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:42.679865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:42.679865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.679865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:42.684832 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:42.685083 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:42.685205 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:42.685381 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:42.685780 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:42.686051 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:42.686311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.686889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:42.686889 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:42.686889 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:42.686889 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:42.686889 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:42.686889 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:42.686889 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.686889 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:42.687615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.687615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:42.687615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:42.687615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:42.687615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.687615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:42.687615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:42.687615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.687615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:42.687615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:42.687615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:42.687615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.687852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:42.687852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:42.687852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:42.687852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:42.687852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:42.687852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:42.687852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:42.687852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:42.687852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:42.687852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:42.687852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:42.687852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:42.687852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:42.688060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.688060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:42.688624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.688624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:42.688624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:42.688624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:42.688624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:42.688624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:42.688624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.688624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:42.688624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:42.688624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.688624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:42.688624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:42.688624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.688624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:42.688624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:42.688624 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:42.688908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:42.688908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:42.688908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.688908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:42.688908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:42.688908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:42.688908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:42.688908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:42.688908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:42.688908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:42.688908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:42.688908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:42.688908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:42.688908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:42.688908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:42.688908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:42.688908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:42.688908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:42.688908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:42.689192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:42.689192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:42.689192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:43.865771 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:43.868264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:43.868264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:43.868264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:43.905026 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:43.905241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:43.905838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:43.905838 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:43.905838 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:43.905838 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:43.905838 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:43.905838 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:43.905838 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:43.905838 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:43.906886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:43.906886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:43.906886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:43.906886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:43.906886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:43.906886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:43.906886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:43.906886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:43.906886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:43.906886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:43.906886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:43.906886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:43.907280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:43.907280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:43.907280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:43.907280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:43.907280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:43.907280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:43.907280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:43.907280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:43.907280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:43.907280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:43.907280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:43.907280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:43.907280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:43.907520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:43.907520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:43.908060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:43.908060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:43.908060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:43.908060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:43.908060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:43.908060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:43.908060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:43.908060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:43.908060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:43.908060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:43.908060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:43.908060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:43.908060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:43.908060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:43.908060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:43.908060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:43.908429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:43.908429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:43.908429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:43.908429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:43.908429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:43.908429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:43.908429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:43.908429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:43.908429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:43.908429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:43.908429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:43.908429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:43.908429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:43.908429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:43.908429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:43.908429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:43.908429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:43.908429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:43.908429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:43.908976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:43.908976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:43.908976 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:47.663850 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:47.664597 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:47.664765 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:47.665688 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:47.665937 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:47.666089 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:47.666246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.666917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:47.666917 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:47.666917 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:47.666917 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:47.666917 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:47.666917 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:47.666917 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.666917 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:47.667635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.667635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:47.667635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:47.667635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:47.667635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.667635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:47.667635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:47.667635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.667635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:47.667635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:47.667635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:47.667635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.668103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:47.668103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:47.668103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:47.668103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:47.668103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:47.668103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:47.668103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:47.668103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:47.668103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:47.668103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:47.668103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:47.668103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:47.668103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:47.669443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.669443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:47.669443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.669443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:47.669443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:47.669443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:47.669443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:47.669443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:47.669443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.669443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:47.669443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:47.669443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.669443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:47.669443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:47.669443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.669921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:47.669921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:47.669921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.669921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:47.669921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:47.669921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.669921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:47.669921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:47.669921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:47.669921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.669921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:47.669921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:47.669921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:47.669921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:47.669921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:47.669921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:47.669921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:47.669921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:47.669921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:47.670550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:47.670550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:47.670550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:47.670550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:47.670550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.670550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:47.670550 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:47.670550 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:47.670550 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:47.670550 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:49:47.670816 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:47.670816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.670816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:47.670816 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:47.670816 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:47.670816 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:47.670816 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:47.670816 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:47.670816 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.670816 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:47.670816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.670816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:47.670816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:47.670816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:47.671063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.671063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:47.671063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:47.671063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:47.671063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.671063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:47.671063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:47.671063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:47.671063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:47.671063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:47.671063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:47.671063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:47.671063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:47.671063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:47.671063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:47.671063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:47.671063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:47.671063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:47.671401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.671401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:47.671439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.671439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:47.671439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:47.671439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:47.671439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.671439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:47.671439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:47.671439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:49:47.671439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:47.671439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:47.671439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.671439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:47.671705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:47.671705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:47.671705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.671705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:47.671705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:47.671705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:47.671705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:47.671705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:47.671705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:47.671705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:47.671705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:47.671705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:47.671705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:47.671705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:47.671921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:47.671921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:47.671921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.671921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:47.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:47.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:47.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:47.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:47.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:47.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:47.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:47.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:47.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:47.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:47.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:47.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.672620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:47.672620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:47.672620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:49:47.672620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:47.672620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:47.672620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.672620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:47.672620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:47.672620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:47.672620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.672620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:47.672620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:47.672620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:47.672620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:47.672620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:47.672620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:47.672620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:47.672620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:47.672984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:47.672984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:47.672984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:47.672984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:47.672984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:47.672984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.672984 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:47.675688 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:47.675967 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:47.676095 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:47.676272 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:47.676739 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:47.676925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.677717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:47.677717 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:47.677717 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:47.677717 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:47.677717 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:47.677717 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:47.677717 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.677717 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:47.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:47.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:47.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:47.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:47.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:47.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:47.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:47.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:47.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:47.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:47.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:47.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:47.679159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:47.679159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:47.679159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:47.679159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:47.679159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:47.679159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:47.679159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:47.679159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.679159 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:47.679789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.679789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:47.679789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:47.679789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:47.679789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.679789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:47.679789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:47.679789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:47.679789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:47.679789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.679789 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:47.680161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:47.680161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:47.680161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.680161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:47.680161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:47.680161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:47.680161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:47.680161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:47.680161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:47.680161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:47.680161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:47.680161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:47.680161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:47.680161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:47.680520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:47.680520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:47.680520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.680520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:47.680927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.680927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:47.680927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:47.680927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:47.680927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:47.680927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:47.680927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.680927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:47.680927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:47.680927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.680927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:47.680927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:47.680927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.680927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:47.680927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:47.680927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.680927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:47.681388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:47.681388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:47.681388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:47.681388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.681388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:47.681388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:47.681388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:47.681388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.681388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:47.681388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:47.681388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:47.681388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:47.681388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:47.681388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:47.681388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:47.681388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:47.681388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:47.681388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:47.681388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:47.681865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:47.681865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:47.681865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.681865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:47.685243 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:47.685556 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:47.685712 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:47.685883 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:47.686262 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:47.686452 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:47.686576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.687030 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:47.687030 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:47.687030 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:47.687030 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:47.687030 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:47.687030 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:47.687030 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.687030 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:47.688026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.688026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:47.688026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:47.688026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:47.688026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.688026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:47.688026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:47.688026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.688026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:47.688026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:47.688026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:47.688026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.688464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:47.688464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:47.688464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:47.688464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:47.688464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:47.688464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:47.688464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:47.688464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:47.688464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:47.688464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:47.688464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:47.688464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:47.688464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:47.688860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.688860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:47.689160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.689160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:47.689160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:47.689160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:47.689160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:47.689160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:47.689160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.689160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:47.689160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:47.689160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.689160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:47.689160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:47.689160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.689160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:47.689160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:47.689160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:47.689554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:47.689554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:47.689554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.689554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:47.689554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:47.689554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:47.689554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:47.689554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:47.689554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:47.689554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:47.689554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:47.689554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:47.689554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:47.689554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:47.689554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:47.689554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:47.689554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:47.689554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:47.689554 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:47.690240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:47.690240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:47.690240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:48.866523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:48.867027 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:48.867063 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:48.867857 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:48.908357 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:48.908843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:48.909544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:48.909544 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:48.909544 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:48.909544 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:48.909544 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:48.909544 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:48.909544 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:48.909544 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:48.910781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:48.910781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:48.910781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:48.910781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:48.910781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:48.910781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:48.910781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:48.910781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:48.910781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:48.910781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:48.910781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:48.910781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:48.911142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:48.911142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:48.911142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:48.911142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:48.911142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:48.911142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:48.911142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:48.911142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:48.911142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:48.911142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:48.911142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:48.911142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:48.911142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:48.911481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:48.911481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:48.911481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:48.911481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:48.911481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:48.911481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:48.911481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:48.911481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:48.911481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:48.911481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:48.911481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:48.911481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:48.911481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:48.911481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:48.911481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:48.911925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:48.911925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:48.911925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:48.911925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:48.911925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:48.911925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:48.911925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:48.911925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:48.911925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:48.911925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:48.911925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:48.911925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:48.911925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:48.911925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:48.911925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:48.911925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:48.911925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:48.911925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:48.911925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:48.912507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:48.912507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:48.912507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:48.912507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:48.912507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:48.912507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:52.664773 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:52.665214 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:52.665342 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:52.666799 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:52.666799 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:52.666799 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:52.666799 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.666959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:52.666959 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:52.666959 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:52.666959 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:52.666959 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:52.666959 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:52.666959 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.666959 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:52.667588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.667588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:52.667588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:52.667588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:52.667588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.667588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:52.667588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:52.667588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.667588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:52.667588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:52.667588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:52.667588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.667890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:52.667890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:52.667890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:52.667890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:52.667890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:52.667890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:52.667890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:52.667890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:52.667890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:52.667890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:52.667890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:52.667890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:52.667890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:52.668201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.668201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:52.668938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.668938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:52.668938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:52.668938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:52.668938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:52.668938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:52.668938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.668938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:52.668938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:52.668938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.668938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:52.668938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:52.668938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.668938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:52.668938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:52.668938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.669540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:52.669540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:52.669540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.669540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:52.669540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:52.669540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:52.669540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.669540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:52.669540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:52.669540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:52.669540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:52.669540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:52.669540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:52.669540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:52.669540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:52.669540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:52.669540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:52.669540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:52.669540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:52.669947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:52.669947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.669947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:52.669947 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:52.669947 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:52.669947 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:52.669947 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:49:52.671116 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:52.671116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.671296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:52.671296 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:52.671296 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:52.671296 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:52.671296 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:52.671296 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:52.671296 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.671296 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:52.671993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.671993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:52.671993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:52.671993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:52.671993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.671993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:52.671993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:52.671993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:52.671993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.671993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:52.671993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:52.671993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:52.671993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:52.671993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:52.671993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:52.672616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:52.672616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:52.672616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:52.672616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:52.672616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:52.672616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:52.672616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:52.672616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.672616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:52.673074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.673074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:52.673074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:52.673074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:52.673074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.673074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:52.673074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:52.673074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:49:52.673074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:52.673074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:52.673074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.673074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:52.673571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:52.673571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:52.673571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.673571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:52.673571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:52.673571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:52.673571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:52.673571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:52.673571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:52.673571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:52.673571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:52.673571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:52.673571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:52.673571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:52.674061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:52.674061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:52.674061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.674061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:52.674061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.674061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:52.674061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:52.674061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:52.674061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:52.674061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:52.674061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.674061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:52.674061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:52.674061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.674288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:52.674288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:52.674288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.674288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:52.674288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:52.674288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.674288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:52.674288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:52.674288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:49:52.674288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:52.674288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:52.674288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.674288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:52.674288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:52.674288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:52.674288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.674288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:52.674288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:52.674288 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:52.674573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:52.674573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:52.674573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:52.674573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:52.674573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:52.674573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:52.674573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:52.674573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:52.674573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:52.674573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:52.674573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.674573 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:52.676208 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:52.676512 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:52.676602 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:52.676750 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:52.677442 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:52.677442 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.677551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:52.677551 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:52.677551 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:52.677551 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:52.677551 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:52.677551 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:52.677551 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.677551 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:52.678124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.678124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:52.678124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:52.678124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:52.678124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.678124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:52.678124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:52.678124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:52.678124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.678124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:52.678124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:52.678124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:52.678124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:52.678124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:52.678124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:52.678454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:52.678454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:52.678454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:52.678454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:52.678454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:52.678454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:52.678454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:52.678454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.678454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:52.678794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.678794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:52.678794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:52.678794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:52.678794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.678794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:52.678794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:52.678794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:52.678794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:52.678794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.678794 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:52.679193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:52.679193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:52.679193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.679193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:52.679193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:52.679193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:52.679193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:52.679193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:52.679193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:52.679193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:52.679193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:52.679193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:52.679193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:52.679193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:52.679733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:52.679733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:52.679733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.679733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:52.679922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.679922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:52.679922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:52.679922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:52.679922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:52.679922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:52.679922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.679922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:52.679922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:52.679922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.679922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:52.679922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:52.679922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.679922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:52.679922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:52.679922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.679922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:52.680231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:52.680231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:52.680231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:52.680231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.680231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:52.680231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:52.680231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:52.680231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.680231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:52.680231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:52.680231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:52.680231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:52.680231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:52.680231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:52.680231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:52.680231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:52.680231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:52.680231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:52.680231 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:52.680521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:52.680521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:52.680521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.680521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:52.685788 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:52.686102 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:52.686209 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:52.686380 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:52.687282 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:52.687282 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:52.687282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.687431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:52.687431 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:52.687431 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:52.687431 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:52.687431 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:52.687431 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:52.687431 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.687431 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:52.688071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.688071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:52.688071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:52.688071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:52.688071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.688071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:52.688071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:52.688071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.688071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:52.688071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:52.688071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:52.688071 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.688498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:52.688498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:52.688498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:52.688498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:52.688498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:52.688498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:52.688498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:52.688498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:52.688498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:52.688498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:52.688498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:52.688498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:52.688498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:52.688942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.688942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:52.689066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.689066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:52.689066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:52.689066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:52.689066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:52.689066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:52.689066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.689066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:52.689066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:52.689066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.689066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:52.689066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:52.689066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.689066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:52.689066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:52.689066 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:52.689491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:52.689491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:52.689491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.689491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:52.689491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:52.689491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:52.689491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:52.689491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:52.689491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:52.689491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:52.689491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:52.689491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:52.689491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:52.689491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:52.689491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:52.689491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:52.689491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:52.689491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:52.689491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:52.689829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:52.689829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:52.689829 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:53.867073 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:53.867532 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:53.867532 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:53.868217 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:53.876608 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=238 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:49:53.912165 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:53.912165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:53.912423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:53.912423 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:53.912423 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:53.912423 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:53.912423 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:53.912423 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:53.912423 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:53.912423 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:53.913438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:53.913438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:53.913438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:53.913438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:53.913438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:53.913438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:53.913438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:53.913438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:53.913438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:53.913438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:53.913438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:53.913438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:53.913710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:53.913710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:53.913710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:53.913710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:53.913710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:53.913710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:53.913710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:53.913710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:53.913710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:53.913710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:53.913710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:53.913710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:53.913710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:53.913952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:53.913952 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:53.914303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:53.914303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:53.914303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:53.914303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:53.914303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:53.914303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:53.914303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:53.914303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:53.914303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:53.914303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:53.914303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:53.914303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:53.914303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:53.914303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:53.914303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:53.914303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:53.914549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:53.914549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:53.914549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:53.914549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:53.914549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:53.914549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:53.914549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:53.914549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:53.914549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:53.914549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:53.914549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:53.914549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:53.914549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:53.914549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:53.914549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:53.914549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:53.914549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:53.914549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:53.914549 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:53.914937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:53.914937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:53.914937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:57.665310 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:57.665821 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:57.665821 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:57.666568 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:57.666848 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:57.666992 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:57.667142 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.667679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:57.667679 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:57.667679 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:57.667679 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:57.667679 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:57.667679 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:57.667679 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.667679 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:57.668750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.668750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:57.668750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:57.668750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:57.668750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.668750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:57.668750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:57.668750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.668750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:57.668750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:57.668750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:57.668750 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.668988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:57.668988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:57.668988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:57.668988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:57.668988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:57.668988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:57.668988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:57.668988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:57.668988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:57.668988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:57.668988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:57.668988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:57.668988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:57.669193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.669193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:57.669901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.669901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:57.669901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:57.669901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:57.669901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:57.669901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:57.669901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.669901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:57.669901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:57.669901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.669901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:57.669901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:57.669901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.669901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:57.669901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:57.669901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.670722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:49:57.670722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:57.670722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.670722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:57.670722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:57.670722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:57.670722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.670722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:57.670722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:57.670722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:57.670722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:57.670722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:57.670722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:57.670722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:57.670722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:57.670722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:57.670722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:57.670722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:57.670722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:57.670990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:57.670990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.670990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:57.670990 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:57.670990 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:57.670990 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:57.670990 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:49:57.670990 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:57.671132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.671395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:57.671395 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:57.671395 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:57.671395 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:57.671395 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:57.671395 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:57.671395 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.671395 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:57.672149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.672149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:57.672149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:57.672149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:57.672149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.672149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:57.672149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:57.672149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:57.672149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.672149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:57.672149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:57.672149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:57.672149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:57.672149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:57.672149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:57.672443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:57.672443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:57.672443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:57.672443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:57.672443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:57.672443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:57.672443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:57.672443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.672443 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:57.673059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.673059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:57.673059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:57.673059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:57.673059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.673059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:57.673059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:57.673059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:49:57.673059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:57.673059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:57.673059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.673059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:57.673276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:57.673276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:57.673276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.673276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:57.673276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:57.673276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:57.673276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:57.673276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:57.673276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:57.673276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:57.673276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:57.673276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:57.673276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:57.673276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:57.673486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:57.673486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:57.673486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.673486 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:57.674170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.674170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:57.674170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:57.674170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:57.674170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:57.674170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:57.674170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.674170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:57.674170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:57.674170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.674170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:57.674170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:57.674170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.674170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:57.674170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:57.674170 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.674472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:49:57.674472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:49:57.674472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:49:57.674472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:57.674472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:57.674472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.674472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:57.674472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:57.674472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:57.674472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.674472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:57.674472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:57.674472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:57.674472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:57.674472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:57.674472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:57.674472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:57.674472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:57.674745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:57.674745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:57.674745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:57.674745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:57.674745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:57.674745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.674745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:57.676570 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:57.676754 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:57.676871 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:57.676999 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:57.677316 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:57.677581 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.678081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:57.678081 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:57.678081 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:57.678081 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:57.678081 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:57.678081 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:57.678081 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.678081 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:57.678795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.678795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:49:57.678795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:57.678795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:57.678795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.678795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:57.678795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:57.678795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:49:57.678795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.678795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:49:57.678795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:57.678795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:49:57.678795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:57.678795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:49:57.678795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:49:57.679169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:57.679169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:49:57.679169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:57.679169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:49:57.679169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:49:57.679169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:57.679169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:49:57.679169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.679169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:49:57.679733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.679733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:57.679733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:57.679733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:57.679733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.679733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:57.679733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:57.679733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:57.679733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:57.679733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.679733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:57.680023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:57.680023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:57.680023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.680023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:57.680023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:57.680023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:57.680023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:57.680023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:57.680023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:57.680023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:57.680023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:57.680023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:57.680023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:57.680023 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:57.680370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:57.680370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:57.680370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.680370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:57.680890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.680890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:57.680890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:57.680890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:57.680890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:57.680890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:57.680890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.680890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:57.680890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:57.680890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.680890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:57.680890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:57.680890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.680890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:57.680890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:57.680890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.680890 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:49:57.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:49:57.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:49:57.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:49:57.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:57.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:57.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:57.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:57.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:57.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:57.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:57.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:57.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:57.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:57.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:57.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:57.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:57.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:57.681655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:57.681655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:57.681655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.681655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:57.686208 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:57.686423 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:57.686554 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:57.686734 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:57.687067 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:57.687343 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:57.687522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.688053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:57.688053 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:57.688053 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:57.688053 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:57.688053 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:57.688053 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:57.688053 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.688053 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:57.688727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.688727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:57.688727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:57.688727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:57.688727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.688727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:57.688727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:57.688727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.688727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:57.688727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:57.688727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:57.688727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.689013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:57.689013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:57.689013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:57.689013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:57.689013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:57.689013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:57.689013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:57.689013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:57.689013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:57.689013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:57.689013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:57.689013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:57.689013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:57.689336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.689336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:57.689336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.689336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:57.689336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:57.689336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:57.689336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:57.689336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:57.689336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.689336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:57.689336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:57.689336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.689336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:57.689336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:57.689336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.689677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:57.689677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:57.689677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:57.689677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:49:57.689677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:57.689677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.689677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:57.689677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:57.689677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:57.689677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:57.689677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:57.689677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:57.689677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:57.689677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:57.689677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:57.689677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:57.689677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:57.689677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:57.689677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:57.690095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:57.690095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:57.690095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:57.690095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:57.690095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:57.690095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:49:58.867541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:49:58.868171 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:49:58.868171 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:49:58.868799 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:49:58.900322 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:49:58.900493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:58.900956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:49:58.900956 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:58.900956 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:49:58.900956 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:49:58.900956 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:58.900956 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:49:58.900956 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:58.900956 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:49:58.901689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:58.901689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:49:58.901689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:58.901689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:49:58.901689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:58.901689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:58.901689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:58.901689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:58.901689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:58.901689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:58.901689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:49:58.901689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:58.901935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:49:58.901935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:58.901935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:49:58.901935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:58.901935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:49:58.901935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:49:58.901935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:58.901935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:49:58.901935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:58.901935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:49:58.901935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:49:58.901935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:58.901935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:49:58.902242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:58.902242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:49:58.902574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:58.902574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:49:58.902574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:49:58.902574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:49:58.902574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:49:58.902574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:49:58.902574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:58.902574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:49:58.902574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:49:58.902574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:58.902574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:49:58.902574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:49:58.902574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:49:58.902574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:49:58.902574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:49:58.902574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:49:58.903448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:49:58.903448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:49:58.903448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:58.903448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:58.903448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:49:58.903448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:49:58.903448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:49:58.903448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:49:58.903448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:49:58.903448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:49:58.903448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:49:58.903448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:49:58.903448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:49:58.903448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:49:58.903448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:49:58.903448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:49:58.903448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:49:58.903448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:49:58.903448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:49:58.903843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:49:58.903843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:49:58.903843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:02.665772 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:02.666168 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:02.666201 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:02.666811 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:02.666951 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:02.667060 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:02.667162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.667592 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:02.667592 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:02.667592 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:02.667592 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:02.667592 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:02.667592 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:02.667592 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.667592 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:02.668311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.668311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:02.668311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:02.668311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:02.668311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.668311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:02.668311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:02.668311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.668311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:02.668311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:02.668311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:02.668311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.668520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:02.668520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:02.668520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:02.668520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:02.668520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:02.668520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:02.668520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:02.668520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:02.668520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:02.668520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:02.668520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:02.668520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:02.668520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:02.668743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.668743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:02.669126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.669126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:02.669126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:02.669126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:02.669126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:02.669126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:02.669126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.669126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:02.669126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:02.669126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.669126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:02.669126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:02.669126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.669126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:02.669126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:02.669126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.669401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:02.669401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:02.669401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.669401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:02.669401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:02.669401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:02.669401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.669401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:02.669401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:02.669401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:02.669401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:02.669401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:02.669401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:02.669401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:02.669401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:02.669401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:02.669401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:02.669401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:02.669401 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:02.669694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:02.669694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.669694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:02.670282 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:02.670465 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:02.670555 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:02.670672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:50:02.670974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:02.671139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.671540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:02.671540 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:02.671540 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:02.671540 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:02.671540 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:02.671540 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:02.671540 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.671540 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:02.672090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.672090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:02.672090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:02.672090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:02.672090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.672090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:02.672090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:02.672090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:02.672090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.672090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:02.672090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:02.672090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:02.672090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:02.672090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:02.672090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:02.672351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:02.672351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:02.672351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:02.672351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:02.672351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:02.672351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:02.672351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:02.672351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.672351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:02.672701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.672701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:02.672701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:02.672701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:02.672701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.672701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:02.672701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:02.672701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:50:02.672701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:02.672701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:02.672701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.672701 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:02.672910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:02.672910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:02.672910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.672910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:02.672910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:02.672910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:02.672910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:02.672910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:02.672910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:02.672910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:02.672910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:02.672910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:02.672910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:02.672910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:02.673110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:02.673110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:02.673110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.673110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:02.673505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.673505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:02.673505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:02.673505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:02.673505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:02.673505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:02.673505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.673505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:02.673505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:02.673505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.673505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:02.673505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:02.673505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.673505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:02.673505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:02.673505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:02.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:02.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:50:02.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:02.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:02.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:02.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:02.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:02.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:02.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:02.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:02.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:02.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:02.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:02.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:02.673778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:02.674038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:02.674038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:02.674038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:02.674038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:02.674038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:02.674038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.674038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:02.676817 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:02.677015 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:02.677117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:02.677232 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:02.677538 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:02.677780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.678201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:02.678201 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:02.678201 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:02.678201 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:02.678201 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:02.678201 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:02.678201 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.678201 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:02.678770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.678770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:02.678770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:02.678770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:02.678770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.678770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:02.678770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:02.678770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:02.678770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.678770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:02.678770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:02.678770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:02.678770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:02.678770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:02.678770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:02.678995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:02.678995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:02.678995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:02.678995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:02.678995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:02.678995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:02.678995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:02.678995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.678995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:02.679427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.679427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:02.679427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:02.679427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:02.679427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.679427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:02.679427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:02.679427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:02.679427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:02.679427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.679427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:02.679613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:02.679613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:02.679613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.679613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:02.679613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:02.679613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:02.679613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:02.679613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:02.679613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:02.679613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:02.679613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:02.679613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:02.679613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:02.679613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:02.679824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:02.679824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:02.679824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.679824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:02.680226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.680226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:02.680226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:02.680226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:02.680226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:02.680226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:02.680226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.680226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:02.680226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:02.680226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.680226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:02.680226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:02.680226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.680226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:02.680226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:02.680226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.680226 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:02.680513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:02.680513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:02.680513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:02.680513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.680513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:02.680513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:02.680513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:02.680513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.680513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:02.680513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:02.680513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:02.680513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:02.680513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:02.680513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:02.680513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:02.680513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:02.680513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:02.680513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:02.680513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:02.680798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:02.680798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:02.680798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.680798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:02.686502 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:02.686744 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:02.686844 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:02.686983 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:02.687265 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:02.687460 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:02.687576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.688038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:02.688038 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:02.688038 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:02.688038 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:02.688038 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:02.688038 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:02.688038 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.688038 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:02.688889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.688889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:02.688889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:02.688889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:02.688889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.688889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:02.688889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:02.688889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.688889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:02.688889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:02.688889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:02.688889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.689097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:02.689097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:02.689097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:02.689097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:02.689097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:02.689097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:02.689097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:02.689097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:02.689097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:02.689097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:02.689097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:02.689097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:02.689097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:02.689314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.689314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:02.689579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.689579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:02.689579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:02.689579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:02.689579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:02.689579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:02.689579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.689579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:02.689579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:02.689579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.689579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:02.689579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:02.689579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.689579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:02.689579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:02.689579 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:02.689843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:02.689843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:02.689843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.689843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:02.689843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:02.689843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:02.689843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:02.689843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:02.689843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:02.689843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:02.689843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:02.689843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:02.689843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:02.689843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:02.689843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:02.689843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:02.689843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:02.689843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:02.689843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:02.690118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:02.690118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:02.690118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:03.868048 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:03.868597 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:03.868667 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:03.869561 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:03.907112 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:03.907402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:03.908095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:03.908095 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:03.908095 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:03.908095 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:03.908095 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:03.908095 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:03.908095 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:03.908095 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:03.909223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:03.909223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:03.909223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:03.909223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:03.909223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:03.909223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:03.909223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:03.909223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:03.909223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:03.909223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:03.909223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:03.909223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:03.909587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:03.909587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:03.909587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:03.909587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:03.909587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:03.909587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:03.909587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:03.909587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:03.909587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:03.909587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:03.909587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:03.909587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:03.909587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:03.909941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:03.909941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:03.910488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:03.910488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:03.910488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:03.910488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:03.910488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:03.910488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:03.910488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:03.910488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:03.910488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:03.910488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:03.910488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:03.910488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:03.910488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:03.910488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:03.910488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:03.910488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:03.910927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:03.910927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:03.910927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:03.910927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:03.910927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:03.910927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:03.910927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:03.910927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:03.910927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:03.910927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:03.910927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:03.910927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:03.910927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:03.910927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:03.910927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:03.910927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:03.910927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:03.910927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:03.910927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:03.911399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:03.911399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:03.911399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:07.666305 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:07.666780 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:07.666780 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:07.667237 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:07.667392 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:07.667507 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:07.667604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.668059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:07.668059 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:07.668059 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:07.668059 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:07.668059 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:07.668059 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:07.668059 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.668059 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:07.668766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.668766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:07.668766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:07.668766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:07.668766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.668766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:07.668766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:07.668766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.668766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:07.668766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:07.668766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:07.668766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.669035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:07.669035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:07.669035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:07.669035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:07.669035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:07.669035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:07.669035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:07.669035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:07.669035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:07.669035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:07.669035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:07.669035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:07.669035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:07.669220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.669220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:07.669545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.669545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:07.669545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:07.669545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:07.669545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:07.669545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:07.669545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.669545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:07.669545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:07.669545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.669545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:07.669545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:07.669545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.669545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:07.669545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:07.669545 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.670140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:07.670140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:07.670140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.670140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:07.670140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:07.670140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:07.670140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.670140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:07.670140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:07.670140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:07.670140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:07.670140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:07.670140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:07.670140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:07.670140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:07.670140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:07.670140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:07.670140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:07.670140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:07.670730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:07.670730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.670730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:07.670730 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:07.670730 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:07.670844 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:07.670866 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:50:07.671159 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:07.671241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.671596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:07.671596 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:07.671596 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:07.671596 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:07.671596 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:07.671596 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:07.671596 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.671596 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:07.672138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.672138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:07.672138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:07.672138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:07.672138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.672138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:07.672138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:07.672138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:07.672138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.672138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:07.672138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:07.672138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:07.672138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:07.672138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:07.672138 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:07.672396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:07.672396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:07.672396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:07.672396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:07.672396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:07.672396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:07.672396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:07.672396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.672396 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:07.672772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.672772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:07.672772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:07.672772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:07.672772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.672772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:07.672772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:07.672772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:50:07.672772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:07.672772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:07.672772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.672772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:07.673035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:07.673035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:07.673035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.673035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:07.673035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:07.673035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:07.673035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:07.673035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:07.673035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:07.673035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:07.673035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:07.673035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:07.673035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:07.673035 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:07.673311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:07.673311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:07.673311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.673311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:07.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:07.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:07.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:07.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:07.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:07.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:07.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:07.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:07.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:07.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:07.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:07.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.673792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:07.673792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:07.673792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:50:07.673792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:07.673792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:07.673792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.673792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:07.673792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:07.673792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:07.673792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.673792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:07.673792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:07.673792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:07.673792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:07.673792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:07.673792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:07.673792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:07.673792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:07.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:07.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:07.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:07.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:07.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:07.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:07.677063 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:07.677283 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:07.677364 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:07.677455 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:07.677721 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:07.677903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.678227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:07.678227 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:07.678227 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:07.678227 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:07.678227 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:07.678227 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:07.678227 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.678227 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:07.678818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.678818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:07.678818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:07.678818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:07.678818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.678818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:07.678818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:07.678818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:07.678818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.678818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:07.678818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:07.678818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:07.678818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:07.678818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:07.678818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:07.679202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:07.679202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:07.679202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:07.679202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:07.679202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:07.679202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:07.679202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:07.679202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.679202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:07.679441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.679441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:07.679441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:07.679441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:07.679441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.679441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:07.679441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:07.679441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:07.679441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:07.679441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.679441 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:07.679720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:07.679720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:07.679720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.679720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:07.679720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:07.679720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:07.679720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:07.679720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:07.679720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:07.679720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:07.679720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:07.679720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:07.679720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:07.679720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:07.680045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:07.680045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:07.680045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.680045 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:07.680372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.680372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:07.680372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:07.680372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:07.680372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:07.680372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:07.680372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.680372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:07.680372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:07.680372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.680372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:07.680372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:07.680372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.680372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:07.680372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:07.680372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.680372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:07.681833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:07.681833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:07.681833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.681833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:07.686837 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:07.687169 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:07.687241 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:07.687389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:07.687652 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:07.687841 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:07.687963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.688393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:07.688393 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:07.688393 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:07.688393 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:07.688393 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:07.688393 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:07.688393 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.688393 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:07.689042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.689042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:07.689042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:07.689042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:07.689042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.689042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:07.689042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:07.689042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.689042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:07.689042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:07.689042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:07.689042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.689683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:07.689683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:07.689683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:07.689683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:07.689683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:07.689683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:07.689683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:07.689683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:07.689683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:07.689683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:07.689683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:07.689683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:07.689683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:07.690038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.690038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:07.690038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.690038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:07.690038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:07.690038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:07.690038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:07.690038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:07.690038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.690038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:07.690038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:07.690038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.690038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:07.690038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:07.690038 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.690404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:07.690404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:07.690404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:07.690404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:07.690404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:07.690404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.690404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:07.690404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:07.690404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:07.690404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:07.690404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:07.690404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:07.690404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:07.690404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:07.690404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:07.690404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:07.690404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:07.690404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:07.690404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:07.691008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:07.691008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:07.691008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:07.691008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:07.691008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:07.691008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:08.868485 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:08.869056 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:08.869056 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:08.869629 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:08.899202 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:08.899487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:08.900131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:08.900131 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:08.900131 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:08.900131 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:08.900131 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:08.900131 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:08.900131 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:08.900131 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:08.900882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:08.900882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:08.900882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:08.900882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:08.900882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:08.900882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:08.900882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:08.900882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:08.900882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:08.900882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:08.900882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:08.900882 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:08.901353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:08.901353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:08.901353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:08.901353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:08.901353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:08.901353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:08.901353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:08.901353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:08.901353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:08.901353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:08.901353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:08.901353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:08.901353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:08.901564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:08.901564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:08.901626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:08.901626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:08.901626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:08.901626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:08.901626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:08.901626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:08.901626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:08.901626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:08.901626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:08.901626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:08.901626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:08.901626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:08.901626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:08.901626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:08.901626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:08.901626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:08.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:08.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:08.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:08.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:08.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:08.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:08.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:08.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:08.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:08.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:08.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:08.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:08.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:08.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:08.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:08.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:08.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:08.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:08.901930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:08.902245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:08.902245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:08.902245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:12.666732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:12.667195 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:12.667228 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:12.667889 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:12.668128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:12.668429 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:12.668558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.669077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:12.669077 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:12.669077 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:12.669077 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:12.669077 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:12.669077 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:12.669077 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.669077 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:12.669832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.669832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:12.669832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:12.669832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:12.669832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.669832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:12.669832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:12.669832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.669832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:12.669832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:12.669832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:12.669832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.670074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:12.670074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:12.670074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:12.670074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:12.670074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:12.670074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:12.670074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:12.670074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:12.670074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:12.670074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:12.670074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:12.670074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:12.670074 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:12.670291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.670291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:12.670584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.670584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:12.670584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:12.670584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:12.670584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:12.670584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:12.670584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.670584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:12.670584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:12.670584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.670584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:12.670584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:12.670584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.670584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:12.670584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:12.670584 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.671435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:12.671435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:12.671435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.671435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:12.671435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:12.671435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:12.671435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.671435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:12.671435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:12.671435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:12.671435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:12.671435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:12.671435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:12.671435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:12.671435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:12.671435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:12.671435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:12.671435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:12.671435 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:12.672029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:12.672029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.672029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:12.672029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:12.672029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:12.672029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:12.672029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:50:12.672029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:12.672174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:12.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:12.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:12.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:12.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:12.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:12.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.672374 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:12.672879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.672879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:12.672879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:12.672879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:12.672879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.672879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:12.672879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:12.672879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:12.672879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.672879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:12.672879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:12.672879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:12.672879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:12.672879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:12.672879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:12.673483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:12.673483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:12.673483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:12.673483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:12.673483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:12.673483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:12.673483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:12.673483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.673483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:12.673483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.673483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:12.673483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:12.673483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:12.673483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.673734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:12.673734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:12.673734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:50:12.673734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:12.673734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:12.673734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.673734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:12.673734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:12.673734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:12.673734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.673734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:12.673734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:12.673734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:12.673925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:12.673925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:12.673925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:12.673925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:12.673925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:12.673925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:12.673925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:12.673925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:12.673925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:12.673925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:12.673925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.673925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:12.674383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.674383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:12.674383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:12.674383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:12.674383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:12.674383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:12.674383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.674383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:12.674383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:12.674383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.674383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:12.674383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:12.674383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.674383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:12.674383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:12.674383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.674815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:12.674815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:12.674815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:50:12.674815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:12.674815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:12.674815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.674815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:12.674815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:12.674815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:12.674815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.674815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:12.674815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:12.674815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:12.674815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:12.674815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:12.674815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:12.674815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:12.674815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:12.675738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:12.675738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:12.675738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:12.675738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:12.675738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:12.675738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.675738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:12.677442 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:12.677736 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:12.677827 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:12.677932 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:12.678264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:12.678475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.678935 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:12.678935 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:12.678935 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:12.678935 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:12.678935 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:12.678935 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:12.678935 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.678935 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:12.679508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.679508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:12.679508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:12.679508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:12.679508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.679508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:12.679508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:12.679508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:12.679508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.679508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:12.679508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:12.679508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:12.679508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:12.679508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:12.679508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:12.679752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:12.679752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:12.679752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:12.679752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:12.679752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:12.679752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:12.679752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:12.679752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.679752 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:12.680105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.680105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:12.680105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:12.680105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:12.680105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.680105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:12.680105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:12.680105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:12.680105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:12.680105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.680105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:12.680517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:12.680517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:12.680517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.680517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:12.680517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:12.680517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:12.680517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:12.680517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:12.680517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:12.680517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:12.680517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:12.680517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:12.680517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:12.680517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:12.680993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:12.680993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:12.680993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.680993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:12.680993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.680993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:12.680993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:12.680993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:12.680993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:12.680993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:12.680993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.680993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:12.680993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:12.680993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.681222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:12.681222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:12.681222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.681222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:12.681222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:12.681222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.681222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:12.681222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:12.681222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:12.681222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:12.681222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.681222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:12.681222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:12.681222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:12.681222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.681222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:12.681222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:12.681222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:12.681222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:12.681524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:12.681524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:12.681524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:12.681524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:12.681524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:12.681524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:12.681524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:12.681524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:12.681524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:12.681524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.681524 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:12.687423 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:12.687679 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:12.687816 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:12.687922 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:12.688417 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:12.688734 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:12.688942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.689483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:12.689483 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:12.689483 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:12.689483 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:12.689483 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:12.689483 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:12.689483 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.689483 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:12.690148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.690148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:12.690148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:12.690148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:12.690148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.690148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:12.690148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:12.690148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.690148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:12.690148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:12.690148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:12.690148 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.690468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:12.690468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:12.690468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:12.690468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:12.690468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:12.690468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:12.690468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:12.690468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:12.690468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:12.690468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:12.690468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:12.690468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:12.690468 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:12.690778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.690778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:12.691067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.691067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:12.691067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:12.691067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:12.691067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:12.691067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:12.691067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.691067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:12.691067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:12.691067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.691067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:12.691067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:12.691067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.691067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:12.691067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:12.691067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:12.691843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:12.691843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:12.691843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.691843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:12.691843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:12.691843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:12.691843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:12.691843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:12.691843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:12.691843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:12.691843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:12.691843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:12.691843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:12.691843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:12.691843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:12.691843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:12.691843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:12.691843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:12.691843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:12.692598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:12.692598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:12.692598 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:13.868996 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:13.869548 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:13.869548 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:13.870248 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:13.898542 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:13.898764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:13.899323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:13.899323 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:13.899323 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:13.899323 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:13.899323 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:13.899323 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:13.899323 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:13.899323 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:13.900078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:13.900078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:13.900078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:13.900078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:13.900078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:13.900078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:13.900078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:13.900078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:13.900078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:13.900078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:13.900078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:13.900078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:13.900347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:13.900347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:13.900347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:13.900347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:13.900347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:13.900347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:13.900347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:13.900347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:13.900347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:13.900347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:13.900347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:13.900347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:13.900347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:13.900681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:13.900681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:13.900896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:13.900896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:13.900896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:13.900896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:13.900896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:13.900896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:13.900896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:13.900896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:13.900896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:13.900896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:13.900896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:13.900896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:13.900896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:13.900896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:13.900896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:13.900896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:13.901144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:13.901144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:13.901144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:13.901144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:13.901144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:13.901144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:13.901144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:13.901144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:13.901144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:13.901144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:13.901144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:13.901144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:13.901144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:13.901144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:13.901144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:13.901144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:13.901144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:13.901144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:13.901144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:13.901464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:13.901464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:13.901464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:17.667319 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:17.667771 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:17.667771 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:17.668382 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:17.668552 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:17.668697 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:17.668811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.669223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:17.669223 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:17.669223 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:17.669223 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:17.669223 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:17.669223 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:17.669223 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.669223 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:17.669896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.669896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:17.669896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:17.669896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:17.669896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.669896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:17.669896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:17.669896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.669896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:17.669896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:17.669896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:17.669896 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.670124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:17.670124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:17.670124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:17.670124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:17.670124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:17.670124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:17.670124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:17.670124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:17.670124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:17.670124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:17.670124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:17.670124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:17.670124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:17.670361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.670361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:17.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:17.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:17.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:17.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:17.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:17.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:17.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:17.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:17.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:17.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:17.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:17.670691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.670953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:17.670953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:17.670953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.670953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:17.670953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:17.670953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:17.670953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.670953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:17.670953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:17.670953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:17.670953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:17.670953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:17.670953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:17.670953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:17.670953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:17.670953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:17.670953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:17.670953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:17.670953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:17.671327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:17.671327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.671327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:17.671327 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:17.671405 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:17.671457 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:17.671546 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:50:17.671841 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:17.671999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.672390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:17.672390 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:17.672390 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:17.672390 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:17.672390 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:17.672390 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:17.672390 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.672390 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:17.673176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.673176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:17.673176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:17.673176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:17.673176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.673176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:17.673176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:17.673176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:17.673176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.673176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:17.673176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:17.673176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:17.673176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:17.673176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:17.673176 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:17.673445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:17.673445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:17.673445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:17.673445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:17.673445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:17.673445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:17.673445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:17.673445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.673445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:17.674090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.674090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:17.674090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:17.674090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:17.674090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.674090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:17.674090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:17.674090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:50:17.674090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:17.674090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:17.674090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.674090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:17.674312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:17.674312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:17.674312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.674312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:17.674312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:17.674312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:17.674312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:17.674312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:17.674312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:17.674312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:17.674312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:17.674312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:17.674312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:17.674312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:17.674541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:17.674541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:17.674541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.674541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:17.675128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.675128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:17.675128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:17.675128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:17.675128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:17.675128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:17.675128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.675128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:17.675128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:17.675128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.675128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:17.675128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:17.675128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.675128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:17.675128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:17.675128 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.675422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:17.675422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:17.675422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:50:17.675422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:17.675422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:17.675422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.675422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:17.675422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:17.675422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:17.675422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.675422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:17.675422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:17.675422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:17.675422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:17.675422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:17.675422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:17.675422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:17.675422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:17.675864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:17.675864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:17.675864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:17.675864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:17.675864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:17.675864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.675864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:17.677861 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:17.678120 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:17.678236 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:17.678367 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:17.678745 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:17.678968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.679427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:17.679427 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:17.679427 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:17.679427 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:17.679427 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:17.679427 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:17.679427 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.679427 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:17.679996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.679996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:17.679996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:17.679996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:17.679996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.679996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:17.679996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:17.679996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:17.679996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.679996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:17.679996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:17.679996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:17.679996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:17.679996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:17.679996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:17.680291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:17.680291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:17.680291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:17.680291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:17.680291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:17.680291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:17.680291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:17.680291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.680291 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:17.680621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.680621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:17.680621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:17.680621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:17.680621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.680621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:17.680621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:17.680621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:17.680621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:17.680621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.680621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:17.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:17.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:17.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:17.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:17.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:17.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:17.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:17.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:17.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:17.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:17.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:17.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:17.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:17.681060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:17.681060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:17.681060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.681060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:17.681380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.681380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:17.681380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:17.681380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:17.681380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:17.681380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:17.681380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.681380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:17.681380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:17.681380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.681380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:17.681380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:17.681380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.681380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:17.681380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:17.681380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.681380 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:17.681699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:17.681699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:17.681699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:17.681699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.681699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:17.681699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:17.681699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:17.681699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.681699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:17.681699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:17.681699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:17.681699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:17.681699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:17.681699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:17.681699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:17.681699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:17.681699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:17.681699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:17.681699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:17.682026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:17.682026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:17.682026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.682026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:17.687825 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:17.688249 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:17.688362 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:17.688507 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:17.688981 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:17.689242 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:17.689414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.690107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:17.690107 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:17.690107 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:17.690107 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:17.690107 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:17.690107 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:17.690107 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.690107 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:17.690852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.690852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:17.690852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:17.690852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:17.690852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.690852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:17.690852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:17.690852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.690852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:17.690852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:17.690852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:17.690852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.691217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:17.691217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:17.691217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:17.691217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:17.691217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:17.691217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:17.691217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:17.691217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:17.691217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:17.691217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:17.691217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:17.691217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:17.691217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:17.691566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.691566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:17.691633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.691633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:17.691633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:17.691633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:17.691633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:17.691633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:17.691633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.691633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:17.691633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:17.691633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.691633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:17.691633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:17.691633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.691633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:17.691633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:17.691633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:17.692154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:17.692154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:17.692154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.692154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:17.692154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:17.692154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:17.692154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:17.692154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:17.692154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:17.692154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:17.692154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:17.692154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:17.692154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:17.692154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:17.692154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:17.692154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:17.692154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:17.692154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:17.692154 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:17.692737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:17.692737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:17.692737 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:18.869465 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:18.870039 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:18.870039 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:18.870706 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:18.903202 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:18.903385 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:18.903838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:18.903838 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:18.903838 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:18.903838 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:18.903838 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:18.903838 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:18.903838 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:18.903838 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:18.904779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:18.904779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:18.904779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:18.904779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:18.904779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:18.904779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:18.904779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:18.904779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:18.904779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:18.904779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:18.904779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:18.904779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:18.905081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:18.905081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:18.905081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:18.905081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:18.905081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:18.905081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:18.905081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:18.905081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:18.905081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:18.905081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:18.905081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:18.905081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:18.905081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:18.905359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:18.905359 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:18.905870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:18.905870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:18.905870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:18.905870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:18.905870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:18.905870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:18.905870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:18.905870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:18.905870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:18.905870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:18.905870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:18.905870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:18.905870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:18.905870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:18.905870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:18.905870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:18.906237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:18.906237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:18.906237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:18.906237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:18.906237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:18.906237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:18.906237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:18.906237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:18.906237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:18.906237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:18.906237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:18.906237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:18.906237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:18.906237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:18.906237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:18.906237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:18.906237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:18.906237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:18.906237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:18.906654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:18.906654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:18.906654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:22.667844 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:22.668474 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:22.668474 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:22.668839 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:22.668978 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:22.669081 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:22.669187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.669641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:22.669641 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:22.669641 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:22.669641 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:22.669641 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:22.669641 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:22.669641 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.669641 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:22.670234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.670234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:22.670234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:22.670234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:22.670234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.670234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:22.670234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:22.670234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.670234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:22.670234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:22.670234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:22.670234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.670639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:22.670639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:22.670639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:22.670639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:22.670639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:22.670639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:22.670639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:22.670639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:22.670639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:22.670639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:22.670639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:22.670639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:22.670639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:22.670995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.670995 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:22.671067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.671067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:22.671067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:22.671067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:22.671067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:22.671067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:22.671067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.671067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:22.671067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:22.671067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.671067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:22.671067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:22.671067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.671067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:22.671067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:22.671067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.671528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:22.671528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:22.671528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.671528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:22.671528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:22.671528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:22.671528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.671528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:22.671528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:22.671528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:22.671528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:22.671528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:22.671528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:22.671528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:22.671528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:22.671528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:22.671528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:22.671528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:22.671528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:22.672085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:22.672085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.672085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:22.672085 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:22.672085 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:22.672085 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:22.672085 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:50:22.672334 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:22.672334 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.672639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:22.672639 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:22.672639 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:22.672639 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:22.672639 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:22.672639 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:22.672639 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.672639 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:22.673277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.673277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:22.673277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:22.673277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:22.673277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.673277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:22.673277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:22.673277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:22.673277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.673277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:22.673277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:22.673277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:22.673277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:22.673277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:22.673277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:22.673727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:22.673727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:22.673727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:22.673727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:22.673727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:22.673727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:22.673727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:22.673727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.673727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:22.673991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.673991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:22.673991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:22.673991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:22.673991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.673991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:22.673991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:22.673991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:50:22.673991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:22.673991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:22.673991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.673991 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:22.674373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:22.674373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:22.674373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.674373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:22.674373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:22.674373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:22.674373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:22.674373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:22.674373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:22.674373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:22.674373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:22.674373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:22.674373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:22.674373 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:22.674769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:22.674769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:22.674769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.674769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:22.674769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.674769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:22.674769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:22.674769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:22.674769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:22.674769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:22.674769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.674769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:22.674769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:22.674769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.675155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:22.675155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:22.675155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.675155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:22.675155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:22.675155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.675155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:22.675155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:22.675155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:50:22.675155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:22.675155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:22.675155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.675155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:22.675155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:22.675155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:22.675155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.675155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:22.675155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:22.675155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:22.675711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:22.675711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:22.675711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:22.675711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:22.675711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:22.675711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:22.675711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:22.675711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:22.675711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:22.675711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:22.675711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.675711 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:22.678222 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:22.678467 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:22.678541 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:22.678639 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:22.678882 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:22.679064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.679431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:22.679431 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:22.679431 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:22.679431 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:22.679431 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:22.679431 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:22.679431 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.679431 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:22.680072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.680072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:22.680072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:22.680072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:22.680072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.680072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:22.680072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:22.680072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:22.680072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.680072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:22.680072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:22.680072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:22.680072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:22.680072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:22.680072 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:22.680497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:22.680497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:22.680497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:22.680497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:22.680497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:22.680497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:22.680497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:22.680497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.680497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:22.680798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.680798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:22.680798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:22.680798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:22.680798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.680798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:22.680798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:22.680798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:22.680798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:22.680798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.680798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:22.681126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:22.681126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:22.681126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.681126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:22.681126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:22.681126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:22.681126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:22.681126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:22.681126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:22.681126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:22.681126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:22.681126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:22.681126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:22.681126 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:22.681487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:22.681487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:22.681487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.681487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:22.681487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.681487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:22.681487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:22.681487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:22.681487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:22.681487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:22.681487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.681487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:22.681487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:22.681487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.681885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:22.681885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:22.681885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.681885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:22.681885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:22.681885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.681885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:22.681885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:22.681885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:22.681885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:22.681885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.681885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:22.681885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:22.681885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:22.681885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.681885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:22.681885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:22.681885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:22.681885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:22.682416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:22.682416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:22.682416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:22.682416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:22.682416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:22.682416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:22.682416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:22.682416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:22.682416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:22.682416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.682416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:22.688346 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:22.688639 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:22.688777 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:22.688936 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:22.689167 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:22.689353 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:22.689464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.689848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:22.689848 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:22.689848 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:22.689848 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:22.689848 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:22.689848 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:22.689848 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.689848 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:22.690637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.690637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:22.690637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:22.690637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:22.690637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.690637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:22.690637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:22.690637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.690637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:22.690637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:22.690637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:22.690637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.691015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:22.691015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:22.691015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:22.691015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:22.691015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:22.691015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:22.691015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:22.691015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:22.691015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:22.691015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:22.691015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:22.691015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:22.691015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:22.691383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.691383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:22.691577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.691577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:22.691577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:22.691577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:22.691577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:22.691577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:22.691577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.691577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:22.691577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:22.691577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.691577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:22.691577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:22.691577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.691577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:22.691577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:22.691577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:22.692039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:22.692039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:22.692039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.692039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:22.692039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:22.692039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:22.692039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:22.692039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:22.692039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:22.692039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:22.692039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:22.692039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:22.692039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:22.692039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:22.692039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:22.692039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:22.692039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:22.692039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:22.692039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:22.692562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:22.692562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:22.692562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:23.869951 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:23.870551 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:23.870551 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:23.871407 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:23.898009 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:23.898193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:23.898657 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:23.898657 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:23.898657 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:23.898657 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:23.898657 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:23.898657 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:23.898657 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:23.898657 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:23.899352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:23.899352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:23.899352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:23.899352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:23.899352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:23.899352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:23.899352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:23.899352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:23.899352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:23.899352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:23.899352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:23.899352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:23.899599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:23.899599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:23.899599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:23.899599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:23.899599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:23.899599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:23.899599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:23.899599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:23.899599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:23.899599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:23.899599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:23.899599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:23.899599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:23.899830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:23.899830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:23.900061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:23.900061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:23.900061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:23.900061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:23.900061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:23.900061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:23.900061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:23.900061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:23.900061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:23.900061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:23.900061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:23.900061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:23.900061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:23.900061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:23.900061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:23.900061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:23.900351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:23.900351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:23.900351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:23.900351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:23.900351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:23.900351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:23.900351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:23.900351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:23.900351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:23.900351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:23.900351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:23.900351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:23.900351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:23.900351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:23.900351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:23.900351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:23.900351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:23.900351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:23.900351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:23.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:23.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:23.900697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:27.668320 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:27.668809 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:27.668809 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:27.669389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:27.669600 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:27.669789 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:27.669951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.670532 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:27.670532 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:27.670532 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:27.670532 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:27.670532 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:27.670532 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:27.670532 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.670532 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:27.671431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.671431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:27.671431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:27.671431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:27.671431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.671431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:27.671431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:27.671431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.671431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:27.671431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:27.671431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:27.671431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.671716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:27.671716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:27.671716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:27.671716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:27.671716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:27.671716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:27.671716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:27.671716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:27.671716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:27.671716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:27.671716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:27.671716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:27.671716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:27.671922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.671922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:27.672389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.672389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:27.672389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:27.672389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:27.672389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:27.672389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:27.672389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.672389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:27.672389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:27.672389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.672389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:27.672389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:27.672389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.672389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:27.672389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:27.672389 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:27.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:27.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:27.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:27.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:27.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:27.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:27.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:27.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:27.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:27.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:27.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:27.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:27.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:27.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:27.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:27.672682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:27.673028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:27.673028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.673028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:27.673028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:27.673028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:27.673028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:27.673028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:50:27.673443 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:27.673502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.673983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:27.673983 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:27.673983 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:27.673983 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:27.673983 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:27.673983 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:27.673983 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.673983 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:27.674500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.674500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:27.674500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:27.674500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:27.674500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.674500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:27.674500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:27.674500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:27.674500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.674500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:27.674500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:27.674500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:27.674500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:27.674500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:27.674500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:27.675055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:27.675055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:27.675055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:27.675055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:27.675055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:27.675055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:27.675055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:27.675055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.675055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:27.675055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.675055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:27.675055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:27.675055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:27.675055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.675533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:27.675533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:27.675533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:50:27.675533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:27.675533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:27.675533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.675533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:27.675533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:27.675533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:27.675533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.675533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:27.675533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:27.675533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:27.675939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:27.675939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:27.675939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:27.675939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:27.675939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:27.675939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:27.675939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:27.675939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:27.675939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:27.675939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:27.675939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.675939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:27.676120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.676120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:27.676120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:27.676120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:27.676120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:27.676120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:27.676120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.676120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:27.676120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:27.676120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.676120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:27.676120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:27.676120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.676120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:27.676120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:27.676120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.676369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:27.676369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:27.676369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:50:27.676369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:27.676369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:27.676369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.676369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:27.676369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:27.676369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:27.676369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.676369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:27.676369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:27.676369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:27.676369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:27.676369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:27.676369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:27.676369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:27.676369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:27.676662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:27.676662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:27.676662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:27.676662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:27.676662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:27.676662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.676662 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:27.678533 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:27.678774 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:27.678897 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:27.679030 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:27.679386 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:27.679623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.680150 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:27.680150 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:27.680150 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:27.680150 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:27.680150 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:27.680150 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:27.680150 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.680150 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:27.680686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.680686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:27.680686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:27.680686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:27.680686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.680686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:27.680686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:27.680686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:27.680686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.680686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:27.680686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:27.680686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:27.680686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:27.680686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:27.680686 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:27.680936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:27.680936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:27.680936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:27.680936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:27.680936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:27.680936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:27.680936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:27.680936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.680936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:27.681238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.681238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:27.681238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:27.681238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:27.681238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.681238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:27.681238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:27.681238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:27.681238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:27.681238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.681238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:27.681432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:27.681432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:27.681432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.681432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:27.681432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:27.681432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:27.681432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:27.681432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:27.681432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:27.681432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:27.681432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:27.681432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:27.681432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:27.681432 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:27.681691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:27.681691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:27.681691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.681691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:27.681951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.681951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:27.681951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:27.681951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:27.681951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:27.681951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:27.681951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.681951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:27.681951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:27.681951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.681951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:27.681951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:27.681951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.681951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:27.681951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:27.681951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.681951 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:27.682544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:27.682544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:27.682544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:27.682544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.682544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:27.682544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:27.682544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:27.682544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.682544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:27.682544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:27.682544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:27.682544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:27.682544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:27.682544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:27.682544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:27.682544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:27.682544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:27.682544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:27.682544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:27.683206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:27.683206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:27.683206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.683206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:27.688747 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:27.688974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:27.689059 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:27.689195 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:27.689522 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:27.689772 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:27.689932 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.690466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:27.690466 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:27.690466 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:27.690466 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:27.690466 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:27.690466 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:27.690466 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.690466 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:27.691020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.691020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:27.691020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:27.691020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:27.691020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.691020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:27.691020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:27.691020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.691020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:27.691020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:27.691020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:27.691020 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.691232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:27.691232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:27.691232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:27.691232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:27.691232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:27.691232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:27.691232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:27.691232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:27.691232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:27.691232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:27.691232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:27.691232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:27.691232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:27.691483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.691483 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:27.691673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.691673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:27.691673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:27.691673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:27.691673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:27.691673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:27.691673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.691673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:27.691673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:27.691673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.691673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:27.691673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:27.691673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.691673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:27.691673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:27.691673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:27.692191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:27.692191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:27.692191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.692191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:27.692191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:27.692191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:27.692191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:27.692191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:27.692191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:27.692191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:27.692191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:27.692191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:27.692191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:27.692191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:27.692191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:27.692191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:27.692191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:27.692191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:27.692191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:27.692540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:27.692540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:27.692540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:28.870518 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:28.871131 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:28.871131 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:28.871992 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:28.906146 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:28.906423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:28.906985 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:28.906985 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:28.906985 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:28.906985 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:28.906985 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:28.906985 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:28.906985 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:28.906985 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:28.907936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:28.907936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:28.907936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:28.907936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:28.907936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:28.907936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:28.907936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:28.907936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:28.907936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:28.907936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:28.907936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:28.907936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:28.908205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:28.908205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:28.908205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:28.908205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:28.908205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:28.908205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:28.908205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:28.908205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:28.908205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:28.908205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:28.908205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:28.908205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:28.908205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:28.908423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:28.908423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:28.908804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:28.908804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:28.908804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:28.908804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:28.908804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:28.908804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:28.908804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:28.908804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:28.908804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:28.908804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:28.908804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:28.908804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:28.908804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:28.908804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:28.908804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:28.908804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:28.909097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:28.909097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:28.909097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:28.909097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:28.909097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:28.909097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:28.909097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:28.909097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:28.909097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:28.909097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:28.909097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:28.909097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:28.909097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:28.909097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:28.909097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:28.909097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:28.909097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:28.909097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:28.909097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:28.909977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:28.909977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:28.909977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:32.668771 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:32.669231 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:32.669231 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:32.670470 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:32.670766 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:32.670948 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:32.671110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.671756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:32.671756 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:32.671756 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:32.671756 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:32.671756 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:32.671756 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:32.671756 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.671756 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:32.672691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.672691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:32.672691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:32.672691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:32.672691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.672691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:32.672691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:32.672691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.672691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:32.672691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:32.672691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:32.672691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.673044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:32.673044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:32.673044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:32.673044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:32.673044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:32.673044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:32.673044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:32.673044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:32.673044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:32.673044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:32.673044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:32.673044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:32.673044 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:32.673376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.673376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:32.673376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.673376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:32.673376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:32.673376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:32.673376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:32.673376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:32.673376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.673376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:32.673376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:32.673376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.673376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:32.673376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:32.673376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.673788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:32.673788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:32.673788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.673788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:32.673788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:32.673788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.673788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:32.673788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:32.673788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:32.673788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.673788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:32.673788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:32.673788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:32.673788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:32.673788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:32.673788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:32.673788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:32.673788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:32.673788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:32.674232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:32.674232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:32.674232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:32.674232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:32.674232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.674232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:32.674232 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:32.674232 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:32.674232 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:32.674232 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:50:32.674525 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:32.674525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.675434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:32.675434 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:32.675434 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:32.675434 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:32.675434 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:32.675434 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:32.675434 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.675434 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:32.676036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.676036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:32.676036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:32.676036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:32.676036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.676036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:32.676036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:32.676036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:32.676036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.676036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:32.676036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:32.676036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:32.676036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:32.676036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:32.676036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:32.676430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:32.676430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:32.676430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:32.676430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:32.676430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:32.676430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:32.676430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:32.676430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.676430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:32.676883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.676883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:32.676883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:32.676883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:32.676883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.676883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:32.676883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:32.676883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:50:32.676883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:32.676883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:32.676883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.676883 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:32.677181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:32.677181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:32.677181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.677181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:32.677181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:32.677181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:32.677181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:32.677181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:32.677181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:32.677181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:32.677181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:32.677181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:32.677181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:32.677181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:32.677548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:32.677548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:32.677548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.677548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:32.677928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.677928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:32.677928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:32.677928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:32.677928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:32.677928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:32.677928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.677928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:32.677928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:32.677928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.677928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:32.677928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:32.677928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.677928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:32.677928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:32.677928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.678340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:32.678340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:32.678340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:50:32.678340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:32.678340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:32.678340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.678340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:32.678340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:32.678340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:32.678340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.678340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:32.678340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:32.678340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:32.678340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:32.678340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:32.678340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:32.678340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:32.678340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:32.678813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:32.678813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:32.678813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:32.678813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:32.678813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:32.678813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.678813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:32.678813 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:32.679069 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:32.679188 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:32.679352 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:32.679886 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:32.680018 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.680527 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:32.680527 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:32.680527 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:32.680527 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:32.680527 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:32.680527 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:32.680527 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.680527 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:32.681390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.681390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:32.681390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:32.681390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:32.681390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.681390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:32.681390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:32.681390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:32.681390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.681390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:32.681390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:32.681390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:32.681390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:32.681390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:32.681390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:32.681760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:32.681760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:32.681760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:32.681760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:32.681760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:32.681760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:32.681760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:32.681760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.681760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:32.682310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.682310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:32.682310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:32.682310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:32.682310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.682310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:32.682310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:32.682310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:32.682310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:32.682310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.682310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:32.682497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:32.682497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:32.682497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.682497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:32.682497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:32.682497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:32.682497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:32.682497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:32.682497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:32.682497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:32.682497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:32.682497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:32.682497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:32.682497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:32.682747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:32.682747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:32.682747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.682747 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:32.683414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.683414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:32.683414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:32.683414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:32.683414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:32.683414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:32.683414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.683414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:32.683414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:32.683414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.683414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:32.683414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:32.683414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.683414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:32.683414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:32.683414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.683414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:32.683708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:32.683708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:32.683708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:32.683708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.683708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:32.683708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:32.683708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:32.683708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.683708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:32.683708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:32.683708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:32.683708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:32.683708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:32.683708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:32.683708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:32.683708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:32.683708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:32.683708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:32.683708 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:32.683987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:32.683987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:32.683987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.683987 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:32.689221 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:32.689603 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:32.689747 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:32.689934 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:32.690406 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:32.690645 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:32.690822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.691382 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:32.691382 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:32.691382 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:32.691382 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:32.691382 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:32.691382 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:32.691382 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.691382 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:32.692388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.692388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:32.692388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:32.692388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:32.692388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.692388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:32.692388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:32.692388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.692388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:32.692388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:32.692388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:32.692388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.692815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:32.692815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:32.692815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:32.692815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:32.692815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:32.692815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:32.692815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:32.692815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:32.692815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:32.692815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:32.692815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:32.692815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:32.692815 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:32.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:32.693420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.693420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:32.693420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:32.693420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:32.693420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:32.693420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:32.693420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.693420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:32.693420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:32.693420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.693420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:32.693420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:32.693420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.693420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:32.693420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:32.693420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:32.693680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:32.693680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:32.693680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.693680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:32.693680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:32.693680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:32.693680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:32.693680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:32.693680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:32.693680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:32.693680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:32.693680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:32.693680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:32.693680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:32.693680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:32.693680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:32.693680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:32.693680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:32.693680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:32.694017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:32.694017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:32.694017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:33.871000 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:33.871504 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:33.871636 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:33.872478 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:33.902335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:33.902590 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:33.903266 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:33.903266 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:33.903266 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:33.903266 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:33.903266 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:33.903266 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:33.903266 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:33.903266 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:33.904061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:33.904061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:33.904061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:33.904061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:33.904061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:33.904061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:33.904061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:33.904061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:33.904061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:33.904061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:33.904061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:33.904061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:33.904681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:33.904681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:33.904681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:33.904681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:33.904681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:33.904681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:33.904681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:33.904681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:33.904681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:33.904681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:33.904681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:33.904681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:33.904681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:33.905014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:33.905014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:33.905014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:33.905014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:33.905014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:33.905014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:33.905014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:33.905014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:33.905014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:33.905014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:33.905014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:33.905014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:33.905014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:33.905014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:33.905014 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:33.905301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:33.905301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:33.905301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:33.905301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:33.905301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:33.905301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:33.905301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:33.905301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:33.905301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:33.905301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:33.905301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:33.905301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:33.905301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:33.905301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:33.905301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:33.905301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:33.905301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:33.905301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:33.905301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:33.905681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:33.905681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:33.905681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:33.905681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:33.905681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:33.905681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:37.669264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:37.669736 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:37.669736 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:37.670356 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:37.670521 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:37.670660 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:37.670812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.671352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:37.671352 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:37.671352 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:37.671352 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:37.671352 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:37.671352 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:37.671352 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.671352 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:37.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:37.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:37.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:37.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:37.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:37.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:37.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:37.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:37.672242 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.672504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:37.672504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:37.672504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:37.672504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:37.672504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:37.672504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:37.672504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:37.672504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:37.672504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:37.672504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:37.672504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:37.672504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:37.672504 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:37.672734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.672734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:37.673173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.673173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:37.673173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:37.673173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:37.673173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:37.673173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:37.673173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.673173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:37.673173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:37.673173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.673173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:37.673173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:37.673173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.673173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:37.673173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:37.673173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.673495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:37.673495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:37.673495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.673495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:37.673495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:37.673495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:37.673495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.673495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:37.673495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:37.673495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:37.673495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:37.673495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:37.673495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:37.673495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:37.673495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:37.673495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:37.673495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:37.673495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:37.673495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:37.673810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:37.673810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.673810 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:37.673810 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:37.673892 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:37.674024 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:37.674162 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:50:37.674491 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:37.674682 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.675079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:37.675079 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:37.675079 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:37.675079 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:37.675079 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:37.675079 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:37.675079 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.675079 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:37.675751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.675751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:37.675751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:37.675751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:37.675751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.675751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:37.675751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:37.675751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:37.675751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.675751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:37.675751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:37.675751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:37.675751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:37.675751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:37.675751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:37.676055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:37.676055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:37.676055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:37.676055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:37.676055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:37.676055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:37.676055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:37.676055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.676055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:37.676528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.676528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:37.676528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:37.676528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:37.676528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.676528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:37.676528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:37.676528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:50:37.676528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:37.676528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:37.676528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.676528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:37.676760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:37.676760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:37.676760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.676760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:37.676760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:37.676760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:37.676760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:37.676760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:37.676760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:37.676760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:37.676760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:37.676760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:37.676760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:37.676760 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:37.676983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:37.676983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:37.676983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.676983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:37.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:37.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:37.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:37.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:37.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:37.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:37.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:37.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:37.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:37.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:37.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:37.677550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.677836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:37.677836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:37.677836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:50:37.677836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:37.677836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:37.677836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.677836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:37.677836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:37.677836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:37.677836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.677836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:37.677836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:37.677836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:37.677836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:37.677836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:37.677836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:37.677836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:37.677836 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:37.678130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:37.678130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:37.678130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:37.678130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:37.678130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:37.678130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.678130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:37.679098 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:37.679317 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:37.679431 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:37.679559 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:37.679838 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:37.680016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.680414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:37.680414 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:37.680414 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:37.680414 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:37.680414 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:37.680414 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:37.680414 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.680414 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:37.680903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.680903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:37.680903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:37.680903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:37.680903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.680903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:37.680903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:37.680903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:37.680903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.680903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:37.680903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:37.680903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:37.680903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:37.680903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:37.680903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:37.681156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:37.681156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:37.681156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:37.681156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:37.681156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:37.681156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:37.681156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:37.681156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.681156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:37.681646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.681646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:37.681646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:37.681646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:37.681646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.681646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:37.681646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:37.681646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:37.681646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:37.681646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.681646 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:37.681867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:37.681867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:37.681867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.681867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:37.681867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:37.681867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:37.681867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:37.681867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:37.681867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:37.681867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:37.681867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:37.681867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:37.681867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:37.681867 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:37.682090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:37.682090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:37.682090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.682090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:37.682603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.682603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:37.682603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:37.682603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:37.682603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:37.682603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:37.682603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.682603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:37.682603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:37.682603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.682603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:37.682603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:37.682603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.682603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:37.682603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:37.682603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.682603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:37.682897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:37.682897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:37.682897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:37.682897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.682897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:37.682897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:37.682897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:37.682897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.682897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:37.682897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:37.682897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:37.682897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:37.682897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:37.682897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:37.682897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:37.682897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:37.682897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:37.682897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:37.682897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:37.683184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:37.683184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:37.683184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.683184 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:37.689585 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:37.689809 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:37.689896 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:37.690033 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:37.690306 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:37.690529 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:37.690690 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.691151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:37.691151 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:37.691151 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:37.691151 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:37.691151 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:37.691151 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:37.691151 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.691151 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:37.691833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.691833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:37.691833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:37.691833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:37.691833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.691833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:37.691833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:37.691833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.691833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:37.691833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:37.691833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:37.691833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.692061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:37.692061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:37.692061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:37.692061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:37.692061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:37.692061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:37.692061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:37.692061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:37.692061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:37.692061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:37.692061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:37.692061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:37.692061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:37.692273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.692273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:37.692533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.692533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:37.692533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:37.692533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:37.692533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:37.692533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:37.692533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.692533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:37.692533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:37.692533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.692533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:37.692533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:37.692533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.692533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:37.692533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:37.692533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:37.692818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:37.692818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:37.692818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.692818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:37.692818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:37.692818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:37.692818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:37.692818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:37.692818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:37.692818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:37.692818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:37.692818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:37.692818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:37.692818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:37.692818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:37.692818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:37.692818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:37.692818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:37.692818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:37.693102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:37.693102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:37.693102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:38.871514 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:38.872043 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:38.872043 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:38.872672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:38.905800 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:38.905800 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:38.905945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:38.905945 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:38.905945 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:38.905945 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:38.905945 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:38.905945 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:38.905945 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:38.905945 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:38.906720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:38.906720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:38.906720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:38.906720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:38.906720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:38.906720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:38.906720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:38.906720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:38.906720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:38.906720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:38.906720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:38.906720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:38.907095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:38.907095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:38.907095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:38.907095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:38.907095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:38.907095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:38.907095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:38.907095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:38.907095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:38.907095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:38.907095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:38.907095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:38.907095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:38.907463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:38.907463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:38.907878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:38.907878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:38.907878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:38.907878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:38.907878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:38.907878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:38.907878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:38.907878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:38.907878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:38.907878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:38.907878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:38.907878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:38.907878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:38.907878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:38.907878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:38.907878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:38.908402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:38.908402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:38.908402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:38.908402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:38.908402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:38.908402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:38.908402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:38.908402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:38.908402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:38.908402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:38.908402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:38.908402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:38.908402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:38.908402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:38.908402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:38.908402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:38.908402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:38.908402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:38.908402 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:38.908898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:38.908898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:38.908898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:42.669713 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:42.670191 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:42.670191 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:42.671329 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:42.671329 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:42.671329 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:42.671329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:42.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:42.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:42.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:42.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:42.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:42.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:42.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:42.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:42.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:42.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:42.673498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:42.673863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.673863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:42.673863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:42.673863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:42.673863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.673863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:42.673863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:42.673863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:42.673863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:42.673863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:42.673863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:42.673863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:42.673863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:42.673863 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:42.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:42.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:42.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:42.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:42.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:42.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:42.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:42.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:42.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:42.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:42.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.674093 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:42.674412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:42.674748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:42.674748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:42.674748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:42.674748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:42.674748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:42.674748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:42.674748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:42.674748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:42.674748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.674748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:42.674748 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:42.675028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:42.675145 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:42.675275 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:50:42.675596 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:42.675804 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.676232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:42.676232 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:42.676232 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:42.676232 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:42.676232 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:42.676232 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:42.676232 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.676232 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:42.676758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.676758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:42.676758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:42.676758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:42.676758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.676758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:42.676758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:42.676758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:42.676758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.676758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:42.676758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:42.676758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:42.676758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:42.676758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:42.676758 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:42.677060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:42.677060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:42.677060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:42.677060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:42.677060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:42.677060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:42.677060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:42.677060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.677060 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:42.677387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.677387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:42.677387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:42.677387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:42.677387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.677387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:42.677387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:42.677387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:50:42.677387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:42.677387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:42.677387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.677387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:42.677629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:42.677629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:42.677629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.677629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:42.677629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:42.677629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:42.677629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:42.677629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:42.677629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:42.677629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:42.677629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:42.677629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:42.677629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:42.677629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:42.677900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:42.677900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:42.677900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.677900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:42.678123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.678123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:42.678123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:42.678123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:42.678123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:42.678123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:42.678123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.678123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:42.678123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:42.678123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.678123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:42.678123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:42.678123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.678123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:42.678123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:42.678123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.678416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:42.678416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:42.678416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:50:42.678416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:42.678416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:42.678416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.678416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:42.678416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:42.678416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:42.678416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.678416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:42.678416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:42.678416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:42.678416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:42.678416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:42.678416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:42.678416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:42.678416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:42.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:42.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:42.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:42.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:42.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:42.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.678739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:42.679342 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:42.679516 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:42.679600 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:42.679725 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:42.679972 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:42.680131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.680488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:42.680488 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:42.680488 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:42.680488 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:42.680488 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:42.680488 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:42.680488 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.680488 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:42.681054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.681054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:42.681054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:42.681054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:42.681054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.681054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:42.681054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:42.681054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:42.681054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.681054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:42.681054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:42.681054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:42.681054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:42.681054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:42.681054 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:42.681393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:42.681393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:42.681393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:42.681393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:42.681393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:42.681393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:42.681393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:42.681393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.681393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:42.681733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.681733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:42.681733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:42.681733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:42.681733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.681733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:42.681733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:42.681733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:42.681733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:42.681733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.681733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:42.681983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:42.681983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:42.681983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.681983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:42.681983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:42.681983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:42.681983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:42.681983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:42.681983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:42.681983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:42.681983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:42.681983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:42.681983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:42.681983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:42.682217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:42.682217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:42.682217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.682217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:42.682505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.682505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:42.682505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:42.682505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:42.682505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:42.682505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:42.682505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.682505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:42.682505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:42.682505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.682505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:42.682505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:42.682505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.682505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:42.682505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:42.682505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.682505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:42.682813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:42.682813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:42.682813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:42.682813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.682813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:42.682813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:42.682813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:42.682813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.682813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:42.682813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:42.682813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:42.682813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:42.682813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:42.682813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:42.682813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:42.682813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:42.682813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:42.682813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:42.682813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:42.683144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:42.683144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:42.683144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.683144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:42.689856 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:42.690042 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:42.690131 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:42.690259 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:42.690514 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:42.690728 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:42.690854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.691262 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:42.691262 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:42.691262 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:42.691262 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:42.691262 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:42.691262 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:42.691262 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.691262 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:42.691901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.691901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:42.691901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:42.691901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:42.691901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.691901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:42.691901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:42.691901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.691901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:42.691901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:42.691901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:42.691901 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.692152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:42.692152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:42.692152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:42.692152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:42.692152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:42.692152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:42.692152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:42.692152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:42.692152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:42.692152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:42.692152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:42.692152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:42.692152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:42.692381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.692381 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:42.692716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.692716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:42.692716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:42.692716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:42.692716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:42.692716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:42.692716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.692716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:42.692716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:42.692716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.692716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:42.692716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:42.692716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.692716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:42.692716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:42.692716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:42.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:42.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:42.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:42.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:42.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:42.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:42.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:42.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:42.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:42.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:42.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:42.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:42.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:42.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:42.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:42.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:42.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:42.693144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:42.693488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:42.693488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:42.693488 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:43.871921 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:43.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:43.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:43.874376 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:43.916474 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:43.916474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:43.916474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:43.916474 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:43.916474 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:43.916474 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:43.916474 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:43.916474 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:43.916474 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:43.916474 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:43.916474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:43.916474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:43.916474 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:43.916819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:43.916819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:43.916819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:43.916819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:43.916819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:43.916819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:43.916819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:43.916819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:43.916819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:43.916819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:43.916819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:43.916819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:43.916819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:43.916819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:43.917104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:43.917104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:43.917104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:43.917104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:43.917104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:43.917104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:43.917104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:43.917104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:43.917104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:43.917104 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:43.917324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:43.917324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:43.917324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:43.917324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:43.917324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:43.917324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:43.917324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:43.917324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:43.917324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:43.917324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:43.917324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:43.917324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:43.917324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:43.917324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:43.917324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:43.917324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:43.918000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:43.918000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:43.918000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:43.918000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:43.918000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:43.918000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:43.918000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:43.918000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:43.918000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:43.918000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:43.918000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:43.918000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:43.918000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:43.918000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:43.918000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:43.918000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:43.918000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:43.918000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:43.918000 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:43.918788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:43.918788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:43.918788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:47.670128 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:47.670816 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:47.670816 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:47.671123 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:47.671292 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:47.671418 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:47.671535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.672009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:47.672009 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:47.672009 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:47.672009 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:47.672009 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:47.672009 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:47.672009 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.672009 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:47.672902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.672902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:47.672902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:47.672902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:47.672902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.672902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:47.672902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:47.672902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.672902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:47.672902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:47.672902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:47.672902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.673178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:47.673178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:47.673178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:47.673178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:47.673178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:47.673178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:47.673178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:47.673178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:47.673178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:47.673178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:47.673178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:47.673178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:47.673178 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:47.673482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.673482 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:47.674017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.674017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:47.674017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:47.674017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:47.674017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:47.674017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:47.674017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.674017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:47.674017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:47.674017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.674017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:47.674017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:47.674017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.674017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:47.674017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:47.674017 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.674392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:47.674392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:47.674392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.674392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:47.674392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:47.674392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:47.674392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.674392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:47.674392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:47.674392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:47.674392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:47.674392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:47.674392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:47.674392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:47.674392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:47.674392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:47.674392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:47.674392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:47.674392 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:47.674748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:47.674748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.674748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:47.675068 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:47.675272 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:47.675390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:47.675502 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:50:47.675780 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:47.675922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.676302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:47.676302 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:47.676302 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:47.676302 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:47.676302 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:47.676302 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:47.676302 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.676302 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:47.677098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.677098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:47.677098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:47.677098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:47.677098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.677098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:47.677098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:47.677098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:47.677098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.677098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:47.677098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:47.677098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:47.677098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:47.677098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:47.677098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:47.677431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:47.677431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:47.677431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:47.677431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:47.677431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:47.677431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:47.677431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:47.677431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.677431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:47.677904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.677904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:47.677904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:47.677904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:47.677904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.677904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:47.677904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:47.677904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:50:47.677904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:47.677904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:47.677904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.677904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:47.678205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:47.678205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:47.678205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.678205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:47.678205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:47.678205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:47.678205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:47.678205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:47.678205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:47.678205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:47.678205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:47.678205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:47.678205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:47.678205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:47.678509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:47.678509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:47.678509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.678509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:47.678900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.678900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:47.678900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:47.678900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:47.678900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:47.678900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:47.678900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.678900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:47.678900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:47.678900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.678900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:47.678900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:47.678900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.678900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:47.678900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:47.678900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.679201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:47.679201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:47.679201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:50:47.679201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:47.679201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:47.679201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.679201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:47.679201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:47.679201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:47.679201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.679201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:47.679201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:47.679201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:47.679201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:47.679201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:47.679201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:47.679201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:47.679201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:47.679570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:47.679570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:47.679570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:47.679570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:47.679570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:47.679570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.679570 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:47.679570 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:47.679776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:47.679927 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:47.679998 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:47.680247 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:47.680423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.680781 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:47.680781 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:47.680781 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:47.680781 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:47.680781 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:47.680781 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:47.680781 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.680781 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:47.681515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.681515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:47.681515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:47.681515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:47.681515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.681515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:47.681515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:47.681515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:47.681515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.681515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:47.681515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:47.681515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:47.681515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:47.681515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:47.681515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:47.681822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:47.681822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:47.681822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:47.681822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:47.681822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:47.681822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:47.681822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:47.681822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.681822 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:47.682367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.682367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:47.682367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:47.682367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:47.682367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.682367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:47.682367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:47.682367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:47.682367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:47.682367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.682367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:47.682664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:47.682664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:47.682664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.682664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:47.682664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:47.682664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:47.682664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:47.682664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:47.682664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:47.682664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:47.682664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:47.682664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:47.682664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:47.682664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:47.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:47.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:47.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:47.683391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.683391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:47.683391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:47.683391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:47.683391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:47.683391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:47.683391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.683391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:47.683391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:47.683391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.683391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:47.683391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:47.683391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.683391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:47.683391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:47.683391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.683391 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:47.683738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:47.683738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:47.683738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:47.683738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.683738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:47.683738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:47.683738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:47.683738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.683738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:47.683738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:47.683738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:47.683738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:47.683738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:47.683738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:47.683738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:47.683738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:47.683738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:47.683738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:47.683738 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:47.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:47.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:47.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:47.690133 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:47.690351 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:47.690459 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:47.690615 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:47.690864 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:47.691036 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:47.691152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.691531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:47.691531 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:47.691531 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:47.691531 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:47.691531 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:47.691531 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:47.691531 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.691531 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:47.692347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.692347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:47.692347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:47.692347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:47.692347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.692347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:47.692347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:47.692347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.692347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:47.692347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:47.692347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:47.692347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.692641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:47.692641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:47.692641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:47.692641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:47.692641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:47.692641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:47.692641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:47.692641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:47.692641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:47.692641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:47.692641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:47.692641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:47.692641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:47.692893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.692893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:47.693295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.693295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:47.693295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:47.693295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:47.693295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:47.693295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:47.693295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.693295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:47.693295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:47.693295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.693295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:47.693295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:47.693295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.693295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:47.693295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:47.693295 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:47.693602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:47.693602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:47.693602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.693602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:47.693602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:47.693602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:47.693602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:47.693602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:47.693602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:47.693602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:47.693602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:47.693602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:47.693602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:47.693602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:47.693602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:47.693602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:47.693602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:47.693602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:47.693602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:47.693930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:47.693930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:47.693930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:48.873027 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:48.873604 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:48.873604 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:48.874221 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:48.916180 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:48.916377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:48.916831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:48.916831 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:48.916831 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:48.916831 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:48.916831 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:48.916831 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:48.916831 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:48.916831 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:48.917791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:48.917791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:48.917791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:48.917791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:48.917791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:48.917791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:48.917791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:48.917791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:48.917791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:48.917791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:48.917791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:48.917791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:48.918064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:48.918064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:48.918064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:48.918064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:48.918064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:48.918064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:48.918064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:48.918064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:48.918064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:48.918064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:48.918064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:48.918064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:48.918064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:48.918286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:48.918286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:48.918893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:48.918893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:48.918893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:48.918893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:48.918893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:48.918893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:48.918893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:48.918893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:48.918893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:48.918893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:48.918893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:48.918893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:48.918893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:48.918893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:48.918893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:48.918893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:48.919186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:48.919186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:48.919186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:48.919186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:48.919186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:48.919186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:48.919186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:48.919186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:48.919186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:48.919186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:48.919186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:48.919186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:48.919186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:48.919186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:48.919186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:48.919186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:48.919186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:48.919186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:48.919186 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:48.919538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:48.919538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:48.919538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:52.670599 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:52.671065 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:52.671065 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:52.671758 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:52.671940 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:52.672066 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:52.672195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.672734 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:52.672734 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:52.672734 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:52.672734 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:52.672734 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:52.672734 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:52.672734 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.672734 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:52.673675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.673675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:52.673675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:52.673675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:52.673675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.673675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:52.673675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:52.673675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.673675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:52.673675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:52.673675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:52.673675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.673988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:52.673988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:52.673988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:52.673988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:52.673988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:52.673988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:52.673988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:52.673988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:52.673988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:52.673988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:52.673988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:52.673988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:52.673988 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:52.674345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.674345 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:52.674866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.674866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:52.674866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:52.674866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:52.674866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:52.674866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:52.674866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.674866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:52.674866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:52.674866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.674866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:52.674866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:52.674866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.674866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:52.674866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:52.674866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.675276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:52.675276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:52.675276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.675276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:52.675276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:52.675276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:52.675276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.675276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:52.675276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:52.675276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:52.675276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:52.675276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:52.675276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:52.675276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:52.675276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:52.675276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:52.675276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:52.675276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:52.675276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:52.675735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:52.675735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.675735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:52.675735 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:52.675735 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:52.675735 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:52.675905 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:50:52.676297 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:52.676427 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.676874 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:52.676874 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:52.676874 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:52.676874 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:52.676874 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:52.676874 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:52.676874 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.676874 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:52.677553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.677553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:52.677553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:52.677553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:52.677553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.677553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:52.677553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:52.677553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:52.677553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.677553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:52.677553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:52.677553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:52.677553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:52.677553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:52.677553 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:52.677934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:52.677934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:52.677934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:52.677934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:52.677934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:52.677934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:52.677934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:52.677934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.677934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:52.678568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.678568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:52.678568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:52.678568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:52.678568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.678568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:52.678568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:52.678568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:50:52.678568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:52.678568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:52.678568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.678568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:52.678886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:52.678886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:52.678886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.678886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:52.678886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:52.678886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:52.678886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:52.678886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:52.678886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:52.678886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:52.678886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:52.678886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:52.678886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:52.678886 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:52.679240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:52.679240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:52.679240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.679240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:52.679669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.679669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:52.679669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:52.679669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:52.679669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:52.679669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:52.679669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.679669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:52.679669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:52.679669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.679669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:52.679669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:52.679669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.679669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:52.679669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:52.679669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.680505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:52.680505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:52.680505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:50:52.680505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:52.680505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:52.680505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.680505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:52.680505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:52.680505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:52.680505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.680505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:52.680505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:52.680505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:52.680505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:52.680505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:52.680505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:52.680505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:52.680505 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:52.681127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:52.681127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:52.681127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:52.681127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:52.681127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:52.681127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.681127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:52.681127 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:52.681127 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:52.681127 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:52.681588 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:52.681588 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:52.681588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.681588 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:52.681588 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:52.681588 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:52.681588 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:52.681588 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:52.681588 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:52.681588 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.681588 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:52.682053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.682053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:52.682053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:52.682053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:52.682053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.682053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:52.682053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:52.682053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:52.682053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.682053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:52.682053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:52.682053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:52.682053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:52.682053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:52.682053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:52.682422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:52.682422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:52.682422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:52.682422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:52.682422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:52.682422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:52.682422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:52.682422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.682422 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:52.682977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.682977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:52.682977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:52.682977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:52.682977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.682977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:52.682977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:52.682977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:52.682977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:52.682977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.682977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:52.683282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:52.683282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:52.683282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.683282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:52.683282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:52.683282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:52.683282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:52.683282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:52.683282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:52.683282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:52.683282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:52.683282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:52.683282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:52.683282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:52.683596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:52.683596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:52.683596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.683596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:52.684108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.684108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:52.684108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:52.684108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:52.684108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:52.684108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:52.684108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.684108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:52.684108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:52.684108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.684108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:52.684108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:52.684108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.684108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:52.684108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:52.684108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.684108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:52.684419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:52.684419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:52.684419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:52.684419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.684419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:52.684419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:52.684419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:52.684419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.684419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:52.684419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:52.684419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:52.684419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:52.684419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:52.684419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:52.684419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:52.684419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:52.684419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:52.684419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:52.684419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:52.684713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:52.684713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:52.684713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.684713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:52.690473 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:52.690713 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:52.690827 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:52.690993 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:52.691347 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:52.691534 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:52.691664 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.692100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:52.692100 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:52.692100 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:52.692100 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:52.692100 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:52.692100 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:52.692100 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.692100 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:52.692847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.692847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:52.692847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:52.692847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:52.692847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.692847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:52.692847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:52.692847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.692847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:52.692847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:52.692847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:52.692847 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.693146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:52.693146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:52.693146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:52.693146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:52.693146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:52.693146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:52.693146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:52.693146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:52.693146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:52.693146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:52.693146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:52.693146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:52.693146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:52.693466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.693466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:52.693900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.693900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:52.693900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:52.693900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:52.693900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:52.693900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:52.693900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.693900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:52.693900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:52.693900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.693900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:52.693900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:52.693900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.693900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:52.693900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:52.693900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:52.694164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:52.694164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:52.694164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.694164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:52.694164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:52.694164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:52.694164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:52.694164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:52.694164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:52.694164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:52.694164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:52.694164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:52.694164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:52.694164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:52.694164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:52.694164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:52.694164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:52.694164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:52.694164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:52.694454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:52.694454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:52.694454 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:53.873567 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:53.874136 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:53.874136 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:53.874989 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:53.879678 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=239 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:50:53.906935 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:53.907164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:53.907767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:53.907767 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:53.907767 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:53.907767 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:53.907767 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:53.907767 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:53.907767 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:53.907767 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:53.908606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:53.908606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:53.908606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:53.908606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:53.908606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:53.908606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:53.908606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:53.908606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:53.908606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:53.908606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:53.908606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:53.908606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:53.908866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:53.908866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:53.908866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:53.908866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:53.908866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:53.908866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:53.908866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:53.908866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:53.908866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:53.908866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:53.908866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:53.908866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:53.908866 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:53.909112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:53.909112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:53.909452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:53.909452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:53.909452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:53.909452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:53.909452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:53.909452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:53.909452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:53.909452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:53.909452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:53.909452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:53.909452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:53.909452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:53.909452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:53.909452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:53.909452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:53.909452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:53.909923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:53.909923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:53.909923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:53.909923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:53.909923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:53.909923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:53.909923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:53.909923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:53.909923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:53.909923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:53.909923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:53.909923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:53.909923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:53.909923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:53.909923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:53.909923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:53.909923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:53.909923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:53.909923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:53.910508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:53.910508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:53.910508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:57.670967 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:57.671568 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:57.671568 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:57.672175 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:57.672388 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:57.672514 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:57.672625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.673113 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:57.673113 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:57.673113 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:57.673113 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:57.673113 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:57.673113 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:57.673113 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.673113 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:57.673854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.673854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:57.673854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:57.673854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:57.673854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.673854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:57.673854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:57.673854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.673854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:57.673854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:57.673854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:57.673854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.674478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:57.674478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:57.674478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:57.674478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:57.674478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:57.674478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:57.674478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:57.674478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:57.674478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:57.674478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:57.674478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:57.674478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:57.674478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:57.675198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.675198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:57.675198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.675198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:57.675198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:57.675198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:57.675198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:57.675198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:57.675198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.675198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:57.675198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:57.675198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.675198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:57.675198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:57.675198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.675977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:57.675977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:57.675977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.675977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:50:57.675977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:57.675977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.675977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:57.675977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:57.675977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:57.675977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.675977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:57.675977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:57.675977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:57.675977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:57.675977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:57.675977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:57.675977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:57.675977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:57.675977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:57.676641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:57.676641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:57.676641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:57.676641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:57.676641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.676641 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:57.676641 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:57.676641 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:57.676641 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:57.676641 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:50:57.676826 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:57.676826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.676826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:57.676826 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:57.676826 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:57.676826 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:57.676826 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:57.676826 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:57.676826 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.676826 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:57.677323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.677323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:57.677323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:57.677323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:57.677323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.677323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:57.677323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:57.677323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:57.677323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.677323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:57.677323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:57.677323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:57.677323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:57.677323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:57.677323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:57.677577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:57.677577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:57.677577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:57.677577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:57.677577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:57.677577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:57.677577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:57.677577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.677577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:57.677927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.677927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:57.677927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:57.677927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:57.677927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.677927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:57.677927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:57.677927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:50:57.677927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:57.677927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:57.677927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.677927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:57.678511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:57.678511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:57.678511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.678511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:57.678511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:57.678511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:57.678511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:57.678511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:57.678511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:57.678511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:57.678511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:57.678511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:57.678511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:57.678511 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:57.678904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:57.678904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:57.678904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.678904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:57.678904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.678904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:57.678904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:57.678904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:57.678904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:57.678904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:57.678904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.678904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:57.678904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:57.678904 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:57.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:57.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:57.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:57.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:50:57.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:50:57.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:50:57.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:57.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:57.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:57.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:57.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:57.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:57.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:57.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:57.679773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:57.679773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:57.679773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:57.679773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:57.679773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:57.679773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:57.679773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:57.679773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:57.679773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:57.679773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:57.679773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.679773 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:57.680093 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:57.680535 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:57.680535 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:57.680535 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:57.680720 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:57.680852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.681187 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:57.681187 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:57.681187 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:57.681187 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:57.681187 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:57.681187 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:57.681187 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.681187 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:57.681873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.681873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:50:57.681873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:57.681873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:57.681873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.681873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:57.681873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:57.681873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:50:57.681873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.681873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:50:57.681873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:57.681873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:50:57.681873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:57.681873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:50:57.681873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:50:57.682645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:57.682645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:50:57.682645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:57.682645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:50:57.682645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:50:57.682645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:57.682645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:50:57.682645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.682645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:50:57.682645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.682645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:57.682645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:57.682645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:57.682645 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.683481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:57.683481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:57.683481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:57.683481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:57.683481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.683481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:57.683481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:57.683481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:57.683481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.683481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:57.683481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:57.683481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:57.683481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:57.683792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:57.683792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:57.683792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:57.683792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:57.683792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:57.683792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:57.683792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:57.683792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:57.683792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:57.683792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.683792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:57.683792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.683996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:57.683996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:57.683996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:57.683996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:57.683996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:57.683996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.683996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:57.683996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:57.683996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.683996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:57.683996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:57.683996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.683996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:57.683996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:57.683996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.683996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:50:57.683996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:50:57.683996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:50:57.683996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.684309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:57.690812 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:57.691047 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:57.691158 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:57.691308 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:57.691612 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:57.691867 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:57.692042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.692498 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:57.692498 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:57.692498 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:57.692498 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:57.692498 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:57.692498 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:57.692498 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.692498 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:57.693101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.693101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:57.693101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:57.693101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:57.693101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.693101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:57.693101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:57.693101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.693101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:57.693101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:57.693101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:57.693101 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.693349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:57.693349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:57.693349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:57.693349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:57.693349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:57.693349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:57.693349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:57.693349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:57.693349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:57.693349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:57.693349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:57.693349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:57.693349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:57.693547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.693547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:57.693875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.693875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:57.693875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:57.693875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:57.693875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:57.693875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:57.693875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.693875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:57.693875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:57.693875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.693875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:57.693875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:57.693875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.693875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:57.693875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:57.693875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:57.694156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:50:57.694156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:57.694156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.694156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:57.694156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:57.694156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:57.694156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:57.694156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:57.694156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:57.694156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:57.694156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:57.694156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:57.694156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:57.694156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:57.694156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:57.694156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:57.694156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:57.694156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:57.694156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:57.694448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:57.694448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:57.694448 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:58.874199 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:58.874786 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:58.874786 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:58.875440 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:50:58.880063 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:50:58.905672 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:58.905945 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:58.906346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:58.906346 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:58.906346 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:58.906346 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:58.906346 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:58.906346 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:58.906346 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:58.906346 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:58.907029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:58.907029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:58.907029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:58.907029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:50:58.907029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:58.907029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:58.907029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:58.907029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:58.907029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:58.907029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:58.907029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:58.907029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:58.907414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:58.907414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:58.907414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:58.907414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:58.907414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:58.907414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:58.907414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:58.907414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:58.907414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:58.907414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:58.907414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:58.907414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:58.907414 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:58.907854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:58.907854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:58.907854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:58.907854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:50:58.907854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:50:58.907854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:50:58.907854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:50:58.907854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:50:58.907854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:58.907854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:50:58.907854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:50:58.907854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:58.907854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:50:58.907854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:50:58.907854 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:50:58.908195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:50:58.908195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:50:58.908195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:50:58.908195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:50:58.908195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:58.908195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:58.908195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:58.908195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:58.908195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:50:58.908195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:58.908195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:50:58.908195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:58.908195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:50:58.908195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:58.908195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:50:58.908195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:50:58.908195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:58.908195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:50:58.908195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:58.908520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:50:58.908520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:50:58.908520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:58.908520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:50:58.908520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:58.908520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:50:58.908520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:50:58.908520 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:50:58.914277 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 08 14:50:58.914448 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Number of inconsistencies found at create/update: networks=5, security_groups=1, routers=2, router_ports=2 {{(pid=89274) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 14:50:58.914538 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource a137566a-eaf1-4f5d-a192-15d63e6c049a (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:50:58.945959 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-a137566a-eaf1-4f5d-a192-15d63e6c049a, resource={'id': 'a137566a-eaf1-4f5d-a192-15d63e6c049a', 'name': 'private', 'tenant_id': 'ba473288446843d3a29e25267968ba61', 'project_id': 'ba473288446843d3a29e25267968ba61', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['7b3cb50f-b942-4054-b735-5f7be989b8f8', 'ba10f097-6c2b-4188-8d42-e9415eff0039'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:20:42Z', 'updated_at': '2026-06-08T14:20:44Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 26551}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:58.946144 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-a137566a-eaf1-4f5d-a192-15d63e6c049a, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:58.946364 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-c4b70b51-9d46-4e6b-a1e7-b3e6a6b8b666, external_ids_update=None, columns={'tag': 26551}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:58.946477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:58.946894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:58.946894 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:58.946894 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:58.946894 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:58.946894 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:58.946894 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:58.946894 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:58.946894 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:58.947665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource a137566a-eaf1-4f5d-a192-15d63e6c049a (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:58.947665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:58.947665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:50:58.947665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:50:58.947665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:50:58.947665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:50:58.947665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:50:58.947665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:58.947665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:58.947665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:58.947665 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:58.947908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:58.947908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:58.947908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:58.947908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:58.947908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:58.947908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:58.947908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:58.947908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:58.947908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:58.947908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:58.947908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:58.947908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:58.947908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:58.947908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:58.948123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:58.948123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:58.948123 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:58.948123 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource fffd4e9d-b343-4a97-8e5f-4015f805fc46 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:50:58.974075 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, resource={'id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'name': 'public', 'tenant_id': 'ba473288446843d3a29e25267968ba61', 'project_id': 'ba473288446843d3a29e25267968ba61', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['4ee83802-2181-432f-a76d-d3335bbf4e0b', 'b9859e49-8026-492d-a0a4-11326477713e'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-08T14:20:46Z', 'updated_at': '2026-06-08T14:20:52Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:58.974279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:58.974497 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-0817ce61-1c85-4457-911a-0b8fdccf389d, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:58.974659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:58.975063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:58.975063 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:58.975063 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:58.975063 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:58.975063 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:58.975063 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:58.975063 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:58.975063 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:58.975818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource fffd4e9d-b343-4a97-8e5f-4015f805fc46 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:58.975818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:58.975818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:50:58.975818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:50:58.975818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:50:58.975818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:50:58.975818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:50:58.975818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:58.975818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:58.975818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:58.975818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:58.976027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:58.976027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:58.976027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:58.976027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:58.976027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:58.976027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:58.976027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:58.976027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:58.976027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:58.976027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:58.976027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:58.976027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:58.976027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:58.976027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:58.976240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:58.976240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:58.976240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:58.976240 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 73426422-a7fc-44cd-9fc5-cd2d8fa95c38 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:50:59.001169 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-73426422-a7fc-44cd-9fc5-cd2d8fa95c38, resource={'id': '73426422-a7fc-44cd-9fc5-cd2d8fa95c38', 'name': 'shared', 'tenant_id': 'ba473288446843d3a29e25267968ba61', 'project_id': 'ba473288446843d3a29e25267968ba61', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['2b53474a-74e5-4c44-bb1b-86c8215bd7ab'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:21:52Z', 'updated_at': '2026-06-08T14:21:52Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 292}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.001522 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-73426422-a7fc-44cd-9fc5-cd2d8fa95c38, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.001803 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-5c6a587e-b2f4-42c7-9706-8a2e20b0df46, external_ids_update=None, columns={'tag': 292}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.001934 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.002534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:59.002534 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:59.002534 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:59.002534 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:59.002534 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:59.002534 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:59.002534 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.002534 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:59.003697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 73426422-a7fc-44cd-9fc5-cd2d8fa95c38 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.003697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:59.003697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:50:59.003697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:50:59.003697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:50:59.003697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:50:59.003697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:50:59.003697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:59.003697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:59.003697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:59.003697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:59.003914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:59.003914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:59.003914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:59.003914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:59.003914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:59.003914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:59.003914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:59.003914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:59.003914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:59.003914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:59.003914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:59.003914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:59.003914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:59.003914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:59.004185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:59.004185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.004185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:59.004185 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 24b6ed09-1bb3-4254-a10d-828b7d27f5ff (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:50:59.036833 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-24b6ed09-1bb3-4254-a10d-828b7d27f5ff, resource={'id': '24b6ed09-1bb3-4254-a10d-828b7d27f5ff', 'name': 'tempest-TestVolumeBootPattern-626079824-network', 'tenant_id': '2eaebe9864e04d018c4fc6f6678a999e', 'project_id': '2eaebe9864e04d018c4fc6f6678a999e', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['705c8961-baad-406e-ab5c-2201ea407fe2'], 'standard_attr_id': 1681, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:42:04Z', 'updated_at': '2026-06-08T14:42:04Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 2940}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.037008 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-24b6ed09-1bb3-4254-a10d-828b7d27f5ff, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestVolumeBootPattern-626079824-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.037302 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-6e5ddab1-8110-4b81-9897-eecf8025269d, external_ids_update=None, columns={'tag': 2940}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.037417 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.037908 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:59.037908 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:59.037908 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:59.037908 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:59.037908 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:59.037908 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:59.037908 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.037908 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:59.039110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 24b6ed09-1bb3-4254-a10d-828b7d27f5ff (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.039110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:59.039110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:50:59.039110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:50:59.039110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:50:59.039110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:50:59.039110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:50:59.039110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:59.039110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:59.039110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:59.039110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:59.039342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:59.039342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:59.039342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:59.039342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:59.039342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:59.039342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:59.039342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:59.039342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:59.039342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:59.039342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:59.039342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:59.039342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:59.039342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:59.039342 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:59.039547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:59.039547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.039547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:59.039547 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 3eeaf3a4-bc2e-45fa-a8dd-1ed62594f914 (type: networks) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:50:59.070389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-3eeaf3a4-bc2e-45fa-a8dd-1ed62594f914, resource={'id': '3eeaf3a4-bc2e-45fa-a8dd-1ed62594f914', 'name': 'tempest-TestVolumeBootPattern-756877900-network', 'tenant_id': '1156286d14d248bc81f670d575137b7d', 'project_id': '1156286d14d248bc81f670d575137b7d', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['99e3c1b8-f5e7-4f4e-973a-1bf722b09cb1'], 'standard_attr_id': 1693, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T14:42:08Z', 'updated_at': '2026-06-08T14:42:08Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 1580}, resource_type=networks, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.070608 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-3eeaf3a4-bc2e-45fa-a8dd-1ed62594f914, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestVolumeBootPattern-756877900-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.070870 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-4bee6417-69a0-4600-946b-d17332049503, external_ids_update=None, columns={'tag': 1580}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.070997 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.071383 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:59.071383 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:59.071383 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:59.071383 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:59.071383 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:59.071383 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:59.071383 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.071383 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:59.072398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 3eeaf3a4-bc2e-45fa-a8dd-1ed62594f914 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.072398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:59.072398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:50:59.072398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:50:59.072398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:50:59.072398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:50:59.072398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 14:50:59.072398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:59.072398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:59.072398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:59.072398 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:59.072749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:59.072749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:59.072749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:59.072749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:59.072749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:59.072749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:59.072749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:59.072749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:59.072749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:59.072749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:59.072749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:59.072749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:59.072749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:59.072749 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:59.073088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:59.073088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.073088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:59.073088 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource d3d6198b-1f58-4b51-87d4-beac0a708756 (type: security_groups) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:50:59.102228 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.db.ovn_revision_numbers_db [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Successfully bumped revision number for resource d3d6198b-1f58-4b51-87d4-beac0a708756 (type: security_groups) to 7 Jun 08 14:50:59.102425 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 1e6d0d4f-769b-40ee-9f46-dc2dd219205c (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:50:59.164447 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, resource={'id': '1e6d0d4f-769b-40ee-9f46-dc2dd219205c', 'name': 'tempest-TestVolumeBootPattern-626079824-router', 'project_id': '2eaebe9864e04d018c4fc6f6678a999e', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.12'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::309'}], 'enable_snat': True}, 'gw_port_id': '684225f0-1038-4501-9c7d-cb064d811b41', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.12', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::309', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:42:05Z', 'updated_at': '2026-06-08T14:42:07Z', 'revision_number': 3, 'tenant_id': '2eaebe9864e04d018c4fc6f6678a999e'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.164783 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, columns={'external_ids': {'neutron:router_name': 'tempest-TestVolumeBootPattern-626079824-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.164861 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "684225f0-1038-4501-9c7d-cb064d811b41"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.165018 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "684225f0-1038-4501-9c7d-cb064d811b41"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.165130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.165593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:59.165593 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:59.165593 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:59.165593 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:59.165593 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:59.165593 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:59.165593 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.165593 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:59.165744 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router 1e6d0d4f-769b-40ee-9f46-dc2dd219205c. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.166717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 1e6d0d4f-769b-40ee-9f46-dc2dd219205c (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.166717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:59.166717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:50:59.166717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:50:59.166717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:50:59.166717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:50:59.166717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:50:59.166717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:50:59.166717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:50:59.166717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:50:59.166717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:50:59.166939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:50:59.166939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:50:59.166939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:59.166939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:59.166939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:59.166939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:59.166939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:59.166939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:59.166939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:59.166939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:59.166939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:59.166939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:59.166939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:59.166939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:59.167155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:59.167155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:59.167155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:59.167155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:59.167155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:59.167155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:59.167155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:59.167155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.167155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:59.167155 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 77ab8ddd-dee1-41ae-b38b-83c0f140f13d (type: routers) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:50:59.231475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, resource={'id': '77ab8ddd-dee1-41ae-b38b-83c0f140f13d', 'name': 'tempest-TestVolumeBootPattern-756877900-router', 'project_id': '1156286d14d248bc81f670d575137b7d', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.188'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::2eb'}], 'enable_snat': True}, 'gw_port_id': 'affc847c-00fe-402e-9933-672b6b2f715d', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'external_fixed_ips': [{'ip_address': '172.24.5.188', 'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e'}, {'ip_address': '2001:db8::2eb', 'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T14:42:09Z', 'updated_at': '2026-06-08T14:42:10Z', 'revision_number': 3, 'tenant_id': '1156286d14d248bc81f670d575137b7d'}, resource_type=routers, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.231761 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, columns={'external_ids': {'neutron:router_name': 'tempest-TestVolumeBootPattern-756877900-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.231875 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "affc847c-00fe-402e-9933-672b6b2f715d"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.231962 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "affc847c-00fe-402e-9933-672b6b2f715d"')], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.232042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.232457 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:59.232457 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:59.232457 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:59.232457 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:59.232457 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:59.232457 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:59.232457 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.232457 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:59.232594 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Unable to update router 77ab8ddd-dee1-41ae-b38b-83c0f140f13d. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.233572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 77ab8ddd-dee1-41ae-b38b-83c0f140f13d (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.233572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:59.233572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:50:59.233572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:50:59.233572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:50:59.233572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:50:59.233572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 14:50:59.233572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 14:50:59.233572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 14:50:59.233572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 14:50:59.233572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 14:50:59.233779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 14:50:59.233779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 14:50:59.233779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:59.233779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:59.233779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:59.233779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:59.233779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:59.233779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:59.233779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:59.233779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:59.233779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:59.233779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:59.233779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:59.233779 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:59.233980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:59.233980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:59.233980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:59.233980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:59.233980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:59.233980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:59.233980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:59.233980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.233980 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:59.233980 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource 684225f0-1038-4501-9c7d-cb064d811b41 (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:50:59.305523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-684225f0-1038-4501-9c7d-cb064d811b41, resource={'id': '684225f0-1038-4501-9c7d-cb064d811b41', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:93:99:3d', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '1e6d0d4f-769b-40ee-9f46-dc2dd219205c', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1686, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.12'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::309'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:42:05Z', 'updated_at': '2026-06-08T14:42:07Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.305709 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-684225f0-1038-4501-9c7d-cb064d811b41, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::309/64', '172.24.5.12/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.305931 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=684225f0-1038-4501-9c7d-cb064d811b41, lrouter_port=lrp-684225f0-1038-4501-9c7d-cb064d811b41, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.306089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.306528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:59.306528 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:59.306528 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:59.306528 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:59.306528 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:59.306528 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:59.306528 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.306528 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:59.307198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource 684225f0-1038-4501-9c7d-cb064d811b41 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.307198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:59.307198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:50:59.307198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:50:59.307198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:50:59.307198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:50:59.307198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:50:59.307198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:59.307198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:59.307198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:59.307198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:59.307466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:59.307466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:59.307466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:59.307466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:59.307466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:59.307466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:59.307466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:59.307466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:59.307466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:59.307466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:59.307466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:59.307466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:59.307466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:59.307466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:59.307723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:59.307723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.307723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:59.307723 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Fixing resource affc847c-00fe-402e-9933-672b6b2f715d (type: router_ports) at create/update {{(pid=89274) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 14:50:59.380935 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-affc847c-00fe-402e-9933-672b6b2f715d, resource={'id': 'affc847c-00fe-402e-9933-672b6b2f715d', 'name': '', 'network_id': 'fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'project_id': '', 'mac_address': 'fa:16:3e:4d:b8:2c', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '77ab8ddd-dee1-41ae-b38b-83c0f140f13d', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1698, 'fixed_ips': [{'subnet_id': 'b9859e49-8026-492d-a0a4-11326477713e', 'ip_address': '172.24.5.188'}, {'subnet_id': '4ee83802-2181-432f-a76d-d3335bbf4e0b', 'ip_address': '2001:db8::2eb'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002315', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T14:42:09Z', 'updated_at': '2026-06-08T14:42:13Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.381123 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-affc847c-00fe-402e-9933-672b6b2f715d, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': 'b9859e49-8026-492d-a0a4-11326477713e 4ee83802-2181-432f-a76d-d3335bbf4e0b', 'neutron:network_name': 'neutron-fffd4e9d-b343-4a97-8e5f-4015f805fc46', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::2eb/64', '172.24.5.188/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.381356 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=affc847c-00fe-402e-9933-672b6b2f715d, lrouter_port=lrp-affc847c-00fe-402e-9933-672b6b2f715d, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:50:59.381535 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.381950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:50:59.381950 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:59.381950 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:50:59.381950 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:50:59.381950 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:59.381950 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:50:59.381950 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.381950 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:50:59.382656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Maintenance task: Failed to fix resource affc847c-00fe-402e-9933-672b6b2f715d (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.382656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:50:59.382656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 14:50:59.382656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 14:50:59.382656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 14:50:59.382656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 14:50:59.382656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 14:50:59.382656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:50:59.382656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:59.382656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:59.382656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:50:59.382860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:50:59.382860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:50:59.382860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:50:59.382860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:50:59.382860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:50:59.382860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:50:59.382860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:50:59.382860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:50:59.382860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:50:59.382860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:50:59.382860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:50:59.382860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:50:59.382860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:50:59.382860 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:50:59.383057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:50:59.383057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:50:59.383057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:50:59.383057 np0000002315 neutron-ovn-maintenance-worker[89274]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_for_inconsistencies finished in 0.475 seconds Jun 08 14:51:02.671494 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:02.671926 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:02.671926 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:02.672613 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:02.672890 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:02.673084 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:02.673249 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.673877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:02.673877 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:02.673877 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:02.673877 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:02.673877 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:02.673877 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:02.673877 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.673877 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:02.674659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.674659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:02.674659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:02.674659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:02.674659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.674659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:02.674659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:02.674659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.674659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:02.674659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:02.674659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:02.674659 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.674875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:02.674875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:02.674875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:02.674875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:02.674875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:02.674875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:02.674875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:02.674875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:02.674875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:02.674875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:02.674875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:02.674875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:02.674875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:02.675075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.675075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:02.675376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.675376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:02.675376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:02.675376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:02.675376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:02.675376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:02.675376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.675376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:02.675376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:02.675376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.675376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:02.675376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:02.675376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.675376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:02.675376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:02.675376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.675637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:02.675637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:02.675637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.675637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:02.675637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:02.675637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:02.675637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.675637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:02.675637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:02.675637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:02.675637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:02.675637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:02.675637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:02.675637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:02.675637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:02.675637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:02.675637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:02.675637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:02.675637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:02.675956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:02.675956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.675956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:02.675956 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:02.676304 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:02.676304 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:02.676304 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:51:02.676577 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:02.676791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.677198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:02.677198 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:02.677198 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:02.677198 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:02.677198 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:02.677198 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:02.677198 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.677198 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:02.677791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.677791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:02.677791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:02.677791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:02.677791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.677791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:02.677791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:02.677791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:02.677791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.677791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:02.677791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:02.677791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:02.677791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:02.677791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:02.677791 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:02.678136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:02.678136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:02.678136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:02.678136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:02.678136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:02.678136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:02.678136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:02.678136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.678136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:02.678528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.678528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:02.678528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:02.678528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:02.678528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.678528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:02.678528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:02.678528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:51:02.678528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:02.678528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:02.678528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.678528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:02.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:02.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:02.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:02.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:02.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:02.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:02.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:02.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:02.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:02.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:02.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:02.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:02.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:02.679384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:02.679384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:02.679384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.679384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:02.679384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.679384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:02.679384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:02.679384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:02.679384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:02.679384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:02.679384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.679384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:02.679384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:02.679384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.679597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:02.679597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:02.679597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.679597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:02.679597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:02.679597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.679597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:02.679597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:02.679597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:51:02.679597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:02.679597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:02.679597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.679597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:02.679597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:02.679597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:02.679597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.679597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:02.679597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:02.679597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:02.679912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:02.679912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:02.679912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:02.679912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:02.679912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:02.679912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:02.679912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:02.679912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:02.679912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:02.679912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:02.679912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.679912 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:02.680475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:02.680475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:02.680586 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:02.680660 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:02.680978 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:02.681188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.681610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:02.681610 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:02.681610 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:02.681610 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:02.681610 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:02.681610 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:02.681610 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.681610 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:02.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:02.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:02.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:02.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:02.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:02.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:02.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:02.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:02.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:02.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:02.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:02.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:02.682798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:02.682798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:02.682798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:02.682798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:02.682798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:02.682798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:02.682798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:02.682798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.682798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:02.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:02.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:02.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:02.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:02.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:02.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:02.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:02.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.682959 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:02.683122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:02.683122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:02.683122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.683122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:02.683122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:02.683122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:02.683122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:02.683122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:02.683122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:02.683122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:02.683122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:02.683122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:02.683122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:02.683122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:02.683353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:02.683353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:02.683353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.683353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:02.683728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.683728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:02.683728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:02.683728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:02.683728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:02.683728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:02.683728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.683728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:02.683728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:02.683728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.683728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:02.683728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:02.683728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.683728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:02.683728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:02.683728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.683728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:02.684472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:02.684472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:02.684472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:02.684472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.684472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:02.684472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:02.684472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:02.684472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.684472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:02.684472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:02.684472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:02.684472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:02.684472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:02.684472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:02.684472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:02.684472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:02.684472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:02.684472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:02.684472 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:02.685062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:02.685062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:02.685062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.685062 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:02.691069 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:02.691261 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:02.691390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:02.691525 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:02.691841 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:02.692009 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:02.692131 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.692582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:02.692582 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:02.692582 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:02.692582 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:02.692582 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:02.692582 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:02.692582 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.692582 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:02.693204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.693204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:02.693204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:02.693204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:02.693204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.693204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:02.693204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:02.693204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.693204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:02.693204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:02.693204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:02.693204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.693419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:02.693419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:02.693419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:02.693419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:02.693419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:02.693419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:02.693419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:02.693419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:02.693419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:02.693419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:02.693419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:02.693419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:02.693419 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:02.693621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.693621 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:02.693954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.693954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:02.693954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:02.693954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:02.693954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:02.693954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:02.693954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.693954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:02.693954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:02.693954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.693954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:02.693954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:02.693954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.693954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:02.693954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:02.693954 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:02.694197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:02.694197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:02.694197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.694197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:02.694197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:02.694197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:02.694197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:02.694197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:02.694197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:02.694197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:02.694197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:02.694197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:02.694197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:02.694197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:02.694197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:02.694197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:02.694197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:02.694197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:02.694197 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:02.694512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:02.694512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:02.694512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:03.874813 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:03.875538 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:03.875538 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:03.876041 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:03.903912 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:03.904161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:03.904680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:03.904680 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:03.904680 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:03.904680 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:03.904680 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:03.904680 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:03.904680 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:03.904680 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:03.905399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:03.905399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:03.905399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:03.905399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:03.905399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:03.905399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:03.905399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:03.905399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:03.905399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:03.905399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:03.905399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:03.905399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:03.905622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:03.905622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:03.905622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:03.905622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:03.905622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:03.905622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:03.905622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:03.905622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:03.905622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:03.905622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:03.905622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:03.905622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:03.905622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:03.905916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:03.905916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:03.906195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:03.906195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:03.906195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:03.906195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:03.906195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:03.906195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:03.906195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:03.906195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:03.906195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:03.906195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:03.906195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:03.906195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:03.906195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:03.906195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:03.906195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:03.906195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:03.907059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:03.907059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:03.907059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:03.907059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:03.907059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:03.907059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:03.907059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:03.907059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:03.907059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:03.907059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:03.907059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:03.907059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:03.907059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:03.907059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:03.907059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:03.907059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:03.907059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:03.907059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:03.907059 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:03.907770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:03.907770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:03.907770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:07.671997 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:07.672536 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:07.672692 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:07.673478 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:07.673756 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:07.673929 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:07.674061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.674622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:07.674622 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:07.674622 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:07.674622 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:07.674622 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:07.674622 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:07.674622 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.674622 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:07.675576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.675576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:07.675576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:07.675576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:07.675576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.675576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:07.675576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:07.675576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.675576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:07.675576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:07.675576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:07.675576 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.675812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:07.675812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:07.675812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:07.675812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:07.675812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:07.675812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:07.675812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:07.675812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:07.675812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:07.675812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:07.675812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:07.675812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:07.675812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:07.676016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.676016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:07.676593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.676593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:07.676593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:07.676593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:07.676593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:07.676593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:07.676593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.676593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:07.676593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:07.676593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.676593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:07.676593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:07.676593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.676593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:07.676593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:07.676593 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.676884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:07.676884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:07.676884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.676884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:07.676884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:07.676884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:07.676884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.676884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:07.676884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:07.676884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:07.676884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:07.676884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:07.676884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:07.676884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:07.676884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:07.676884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:07.676884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:07.676884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:07.676884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:07.677158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:07.677158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.677158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:07.677158 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:07.677460 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:07.677460 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:07.677547 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:51:07.677937 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:07.678108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.678582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:07.678582 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:07.678582 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:07.678582 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:07.678582 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:07.678582 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:07.678582 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.678582 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:07.679404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.679404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:07.679404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:07.679404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:07.679404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.679404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:07.679404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:07.679404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:07.679404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.679404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:07.679404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:07.679404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:07.679404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:07.679404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:07.679404 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:07.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:07.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:07.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:07.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:07.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:07.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:07.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:07.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:07.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:07.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:07.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:07.680100 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.680487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:07.680487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:07.680487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:51:07.680487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:07.680487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:07.680487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.680487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:07.680487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:07.680487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:07.680487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.680487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:07.680487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:07.680487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.680832 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:07.681149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.681149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:07.681149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:07.681149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:07.681149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:07.681149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:07.681149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.681149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:07.681149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:07.681149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.681149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:07.681149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:07.681149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.681149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:07.681149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:07.681149 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:07.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:07.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:51:07.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:07.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:07.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:07.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:07.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:07.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:07.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:07.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:07.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:07.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:07.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:07.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:07.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:07.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:07.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:07.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:07.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:07.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:07.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:07.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:07.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:07.682048 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:07.682320 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:07.682320 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:07.682320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.682424 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:07.682424 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:07.682424 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:07.682424 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:07.682424 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:07.682424 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:07.682424 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.682424 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:07.683500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.683500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:07.683500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:07.683500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:07.683500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.683500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:07.683500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:07.683500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:07.683500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.683500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:07.683500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:07.683500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:07.683500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:07.683500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:07.683500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:07.684626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:07.684626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:07.684626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:07.684626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:07.684626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:07.684626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:07.684626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:07.684626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.684626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:07.684626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.684626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:07.684626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:07.684626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:07.684626 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.685352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:07.685352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:07.685352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:07.685352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:07.685352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.685352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:07.685352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:07.685352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:07.685352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.685352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:07.685352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:07.685352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:07.685352 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:07.685986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:07.685986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:07.685986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:07.685986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:07.685986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:07.685986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:07.685986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:07.685986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:07.685986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:07.685986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.685986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:07.685986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.686610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:07.686610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:07.686610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:07.686610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:07.686610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:07.686610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.686610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:07.686610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:07.686610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.686610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:07.686610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:07.686610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.686610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:07.686610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:07.686610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.686610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:07.686610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:07.686610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:07.686610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.687655 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:07.691522 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:07.691776 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:07.691900 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:07.692058 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:07.692440 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:07.692657 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:07.692783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.693233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:07.693233 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:07.693233 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:07.693233 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:07.693233 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:07.693233 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:07.693233 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.693233 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:07.694087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.694087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:07.694087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:07.694087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:07.694087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.694087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:07.694087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:07.694087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.694087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:07.694087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:07.694087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:07.694087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.694305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:07.694305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:07.694305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:07.694305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:07.694305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:07.694305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:07.694305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:07.694305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:07.694305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:07.694305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:07.694305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:07.694305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:07.694305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:07.694556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.694556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:07.694914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.694914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:07.694914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:07.694914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:07.694914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:07.694914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:07.694914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.694914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:07.694914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:07.694914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.694914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:07.694914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:07.694914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.694914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:07.694914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:07.694914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:07.695181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:07.695181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:07.695181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.695181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:07.695181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:07.695181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:07.695181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:07.695181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:07.695181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:07.695181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:07.695181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:07.695181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:07.695181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:07.695181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:07.695181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:07.695181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:07.695181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:07.695181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:07.695181 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:07.695566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:07.695566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:07.695566 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:08.875281 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:08.875840 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:08.875840 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:08.876484 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:08.910240 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:08.910460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:08.910963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:08.910963 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:08.910963 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:08.910963 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:08.910963 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:08.910963 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:08.910963 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:08.910963 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:08.911869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:08.911869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:08.911869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:08.911869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:08.911869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:08.911869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:08.911869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:08.911869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:08.911869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:08.911869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:08.911869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:08.911869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:08.912188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:08.912188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:08.912188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:08.912188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:08.912188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:08.912188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:08.912188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:08.912188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:08.912188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:08.912188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:08.912188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:08.912188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:08.912188 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:08.912440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:08.912440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:08.912969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:08.912969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:08.912969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:08.912969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:08.912969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:08.912969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:08.912969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:08.912969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:08.912969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:08.912969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:08.912969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:08.912969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:08.912969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:08.912969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:08.912969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:08.912969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:08.913531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:08.913531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:08.913531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:08.913531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:08.913531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:08.913531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:08.913531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:08.913531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:08.913531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:08.913531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:08.913531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:08.913531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:08.913531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:08.913531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:08.913531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:08.913531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:08.913531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:08.913531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:08.913531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:08.914127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:08.914127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:08.914127 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:12.672453 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:12.673058 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:12.673058 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:12.673496 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:12.673660 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:12.673794 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:12.673923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.674368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:12.674368 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:12.674368 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:12.674368 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:12.674368 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:12.674368 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:12.674368 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.674368 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:12.675053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.675053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:12.675053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:12.675053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:12.675053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.675053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:12.675053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:12.675053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.675053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:12.675053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:12.675053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:12.675053 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.675319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:12.675319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:12.675319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:12.675319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:12.675319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:12.675319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:12.675319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:12.675319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:12.675319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:12.675319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:12.675319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:12.675319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:12.675319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:12.675515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.675515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:12.675786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.675786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:12.675786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:12.675786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:12.675786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:12.675786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:12.675786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.675786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:12.675786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:12.675786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.675786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:12.675786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:12.675786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.675786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:12.675786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:12.675786 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.676397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:12.676397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:12.676397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.676397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:12.676397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:12.676397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:12.676397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.676397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:12.676397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:12.676397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:12.676397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:12.676397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:12.676397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:12.676397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:12.676397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:12.676397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:12.676397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:12.676397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:12.676397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:12.677144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:12.677144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.677144 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:12.677285 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:12.677461 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:12.677539 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:12.677630 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:51:12.677946 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:12.678124 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.678536 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:12.678536 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:12.678536 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:12.678536 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:12.678536 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:12.678536 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:12.678536 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.678536 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:12.679222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.679222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:12.679222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:12.679222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:12.679222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.679222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:12.679222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:12.679222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:12.679222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.679222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:12.679222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:12.679222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:12.679222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:12.679222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:12.679222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:12.679526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:12.679526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:12.679526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:12.679526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:12.679526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:12.679526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:12.679526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:12.679526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.679526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:12.679871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.679871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:12.679871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:12.679871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:12.679871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.679871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:12.679871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:12.679871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:51:12.679871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:12.679871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:12.679871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.679871 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:12.680379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:12.680379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:12.680379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.680379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:12.680379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:12.680379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:12.680379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:12.680379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:12.680379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:12.680379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:12.680379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:12.680379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:12.680379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:12.680379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:12.680839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:12.680839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:12.680839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.680839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:12.680839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.680839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:12.680839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:12.680839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:12.680839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:12.680839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:12.680839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.680839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:12.680839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:12.680839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.681175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:12.681175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:12.681175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.681175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:12.681175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:12.681175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.681175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:12.681175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:12.681175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:51:12.681175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:12.681175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:12.681175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.681175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:12.681175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:12.681175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:12.681175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.681175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:12.681175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:12.681175 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:12.681565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:12.681565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:12.681565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:12.681565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:12.681565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:12.681565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:12.681565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:12.681565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:12.681565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:12.681565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:12.681565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.681565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:12.681565 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:12.681565 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:12.681814 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:12.681814 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:12.682185 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:12.682185 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.682574 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:12.682574 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:12.682574 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:12.682574 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:12.682574 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:12.682574 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:12.682574 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.682574 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:12.683116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.683116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:12.683116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:12.683116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:12.683116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.683116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:12.683116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:12.683116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:12.683116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.683116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:12.683116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:12.683116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:12.683116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:12.683116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:12.683116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:12.683710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:12.683710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:12.683710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:12.683710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:12.683710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:12.683710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:12.683710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:12.683710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.683710 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:12.684057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.684057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:12.684057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:12.684057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:12.684057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.684057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:12.684057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:12.684057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:12.684057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:12.684057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.684057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:12.684324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:12.684324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:12.684324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.684324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:12.684324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:12.684324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:12.684324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:12.684324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:12.684324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:12.684324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:12.684324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:12.684324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:12.684324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:12.684324 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:12.684540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:12.684540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:12.684540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.684540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:12.684970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.684970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:12.684970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:12.684970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:12.684970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:12.684970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:12.684970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.684970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:12.684970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:12.684970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.684970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:12.684970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:12.684970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.684970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:12.684970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:12.684970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.684970 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:12.685241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:12.685241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:12.685241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:12.685241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.685241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:12.685241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:12.685241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:12.685241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.685241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:12.685241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:12.685241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:12.685241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:12.685241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:12.685241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:12.685241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:12.685241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:12.685241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:12.685241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:12.685241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:12.685625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:12.685625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:12.685625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.685625 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:12.691888 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:12.692117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:12.692233 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:12.692410 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:12.692702 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:12.692888 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:12.693040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.693451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:12.693451 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:12.693451 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:12.693451 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:12.693451 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:12.693451 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:12.693451 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.693451 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:12.694091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.694091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:12.694091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:12.694091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:12.694091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.694091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:12.694091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:12.694091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.694091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:12.694091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:12.694091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:12.694091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.694329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:12.694329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:12.694329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:12.694329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:12.694329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:12.694329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:12.694329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:12.694329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:12.694329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:12.694329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:12.694329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:12.694329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:12.694329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:12.694673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.694673 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:12.695004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.695004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:12.695004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:12.695004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:12.695004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:12.695004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:12.695004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.695004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:12.695004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:12.695004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.695004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:12.695004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:12.695004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.695004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:12.695004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:12.695004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:12.695287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:12.695287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:12.695287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.695287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:12.695287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:12.695287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:12.695287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:12.695287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:12.695287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:12.695287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:12.695287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:12.695287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:12.695287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:12.695287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:12.695287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:12.695287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:12.695287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:12.695287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:12.695287 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:12.695580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:12.695580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:12.695580 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:13.875707 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:13.876286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:13.876286 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:13.876761 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:13.904398 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:13.904548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:13.905094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:13.905094 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:13.905094 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:13.905094 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:13.905094 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:13.905094 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:13.905094 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:13.905094 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:13.905728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:13.905728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:13.905728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:13.905728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:13.905728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:13.905728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:13.905728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:13.905728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:13.905728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:13.905728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:13.905728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:13.905728 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:13.906026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:13.906026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:13.906026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:13.906026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:13.906026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:13.906026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:13.906026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:13.906026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:13.906026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:13.906026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:13.906026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:13.906026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:13.906026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:13.906224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:13.906224 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:13.906507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:13.906507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:13.906507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:13.906507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:13.906507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:13.906507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:13.906507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:13.906507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:13.906507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:13.906507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:13.906507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:13.906507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:13.906507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:13.906507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:13.906507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:13.906507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:13.907429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:13.907429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:13.907429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:13.907429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:13.907429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:13.907429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:13.907429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:13.907429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:13.907429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:13.907429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:13.907429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:13.907429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:13.907429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:13.907429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:13.907429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:13.907429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:13.907429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:13.907429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:13.907429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:13.908069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:13.908069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:13.908069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:17.673051 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:17.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:17.673517 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:17.674295 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:17.674295 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:17.674295 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:17.674393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.674740 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:17.674740 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:17.674740 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:17.674740 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:17.674740 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:17.674740 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:17.674740 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.674740 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:17.676303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.676303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:17.676303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:17.676303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:17.676303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.676303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:17.676303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:17.676303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.676303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:17.676303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:17.676303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:17.676303 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.676601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:17.676601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:17.676601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:17.676601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:17.676601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:17.676601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:17.676601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:17.676601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:17.676601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:17.676601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:17.676601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:17.676601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:17.676601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:17.676923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.676923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:17.676923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.676923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:17.676923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:17.676923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:17.676923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:17.676923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:17.676923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.676923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:17.676923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:17.676923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.676923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:17.676923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:17.676923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.677282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:17.677282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:17.677282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.677282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:17.677282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:17.677282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.677282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:17.677282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:17.677282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:17.677282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.677282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:17.677282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:17.677282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:17.677282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:17.677282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:17.677282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:17.677282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:17.677282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:17.677282 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:17.677743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:17.677743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:17.677743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:17.677743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:17.677743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.677743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:17.677743 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:17.677743 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:17.677920 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:17.677920 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:51:17.678660 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:17.678660 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.679004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:17.679004 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:17.679004 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:17.679004 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:17.679004 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:17.679004 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:17.679004 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.679004 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:17.679533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.679533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:17.679533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:17.679533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:17.679533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.679533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:17.679533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:17.679533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:17.679533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.679533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:17.679533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:17.679533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:17.679533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:17.679533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:17.679533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:17.680102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:17.680102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:17.680102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:17.680102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:17.680102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:17.680102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:17.680102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:17.680102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.680102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:17.680102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.680102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:17.680102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:17.680102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:17.680102 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.680606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:17.680606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:17.680606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:51:17.680606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:17.680606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:17.680606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.680606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:17.680606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:17.680606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:17.680606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.680606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:17.680606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:17.680606 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:17.681067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:17.681067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:17.681067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:17.681067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:17.681067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:17.681067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:17.681067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:17.681067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:17.681067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:17.681067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:17.681067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.681067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:17.681663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.681663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:17.681663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:17.681663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:17.681663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:17.681663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:17.681663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.681663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:17.681663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:17.681663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.681663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:17.681663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:17.681663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.681663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:17.681663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:17.681663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.682068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:17.682068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:17.682068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:51:17.682068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:17.682068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:17.682068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.682068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:17.682068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:17.682068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:17.682068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.682068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:17.682068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:17.682068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:17.682068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:17.682068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:17.682068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:17.682068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:17.682068 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:17.682366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:17.682366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:17.682366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:17.682366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:17.682366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:17.682366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.682366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:17.682366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:17.682366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:17.682366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:17.682530 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:17.682530 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:17.682530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.682583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:17.682583 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:17.682583 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:17.682583 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:17.682583 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:17.682583 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:17.682583 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.682583 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:17.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:17.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:17.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:17.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:17.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:17.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:17.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:17.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:17.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:17.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:17.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:17.683070 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:17.683785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:17.683785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:17.683785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:17.683785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:17.683785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:17.683785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:17.683785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:17.683785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.683785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:17.683785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.683785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:17.683785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:17.683785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:17.683785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.684508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:17.684508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:17.684508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:17.684508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:17.684508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.684508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:17.684508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:17.684508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:17.684508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.684508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:17.684508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:17.684508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:17.684508 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:17.684963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:17.684963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:17.684963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:17.684963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:17.684963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:17.684963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:17.684963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:17.684963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:17.684963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:17.684963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.684963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:17.684963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.685167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:17.685167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:17.685167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:17.685167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:17.685167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:17.685167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.685167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:17.685167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:17.685167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.685167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:17.685167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:17.685167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.685167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:17.685167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:17.685167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.685167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:17.685167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:17.685167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:17.685167 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.685449 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:17.692144 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:17.692350 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:17.692454 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:17.692595 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:17.692815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:17.692980 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:17.693087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.693434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:17.693434 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:17.693434 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:17.693434 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:17.693434 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:17.693434 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:17.693434 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.693434 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:17.693975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.693975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:17.693975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:17.693975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:17.693975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.693975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:17.693975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:17.693975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.693975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:17.693975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:17.693975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:17.693975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.694158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:17.694158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:17.694158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:17.694158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:17.694158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:17.694158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:17.694158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:17.694158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:17.694158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:17.694158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:17.694158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:17.694158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:17.694158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:17.694372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.694372 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:17.694635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.694635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:17.694635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:17.694635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:17.694635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:17.694635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:17.694635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.694635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:17.694635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:17.694635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.694635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:17.694635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:17.694635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.694635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:17.694635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:17.694635 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:17.694895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:17.694895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:17.694895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.694895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:17.694895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:17.694895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:17.694895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:17.694895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:17.694895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:17.694895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:17.694895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:17.694895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:17.694895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:17.694895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:17.694895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:17.694895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:17.694895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:17.694895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:17.694895 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:17.695164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:17.695164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:17.695164 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:18.876092 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:18.876732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:18.876732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:18.877247 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:18.902096 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:18.902283 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:18.902765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:18.902765 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:18.902765 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:18.902765 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:18.902765 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:18.902765 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:18.902765 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:18.902765 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:18.903423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:18.903423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:18.903423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:18.903423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:18.903423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:18.903423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:18.903423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:18.903423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:18.903423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:18.903423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:18.903423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:18.903423 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:18.903676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:18.903676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:18.903676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:18.903676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:18.903676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:18.903676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:18.903676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:18.903676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:18.903676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:18.903676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:18.903676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:18.903676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:18.903676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:18.903881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:18.903881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:18.904140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:18.904140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:18.904140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:18.904140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:18.904140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:18.904140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:18.904140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:18.904140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:18.904140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:18.904140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:18.904140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:18.904140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:18.904140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:18.904140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:18.904140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:18.904140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:18.904445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:18.904445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:18.904445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:18.904445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:18.904445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:18.904445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:18.904445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:18.904445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:18.904445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:18.904445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:18.904445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:18.904445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:18.904445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:18.904445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:18.904445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:18.904445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:18.904445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:18.904445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:18.904445 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:18.904780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:18.904780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:18.904780 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:22.673426 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:22.673925 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:22.673925 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:22.674708 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:22.674892 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:22.675018 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:22.675129 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.675733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:22.675733 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:22.675733 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:22.675733 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:22.675733 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:22.675733 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:22.675733 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.675733 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:22.676795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.676795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:22.676795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:22.676795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:22.676795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.676795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:22.676795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:22.676795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.676795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:22.676795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:22.676795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:22.676795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.677063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:22.677063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:22.677063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:22.677063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:22.677063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:22.677063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:22.677063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:22.677063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:22.677063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:22.677063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:22.677063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:22.677063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:22.677063 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:22.677273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.677273 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:22.678002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.678002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:22.678002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:22.678002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:22.678002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:22.678002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:22.678002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.678002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:22.678002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:22.678002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.678002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:22.678002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:22.678002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.678002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:22.678002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:22.678002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.678400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:22.678400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:22.678400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.678400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:22.678400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:22.678400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:22.678400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.678400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:22.678400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:22.678400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:22.678400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:22.678400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:22.678400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:22.678400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:22.678400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:22.678400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:22.678400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:22.678400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:22.678400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:22.678717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:22.678717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.678717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:22.678717 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:22.678717 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:22.678717 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:22.678855 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:51:22.679337 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:22.679440 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.679950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:22.679950 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:22.679950 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:22.679950 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:22.679950 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:22.679950 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:22.679950 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.679950 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:22.680846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.680846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:22.680846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:22.680846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:22.680846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.680846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:22.680846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:22.680846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:22.680846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.680846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:22.680846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:22.680846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:22.680846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:22.680846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:22.680846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:22.681552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:22.681552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:22.681552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:22.681552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:22.681552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:22.681552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:22.681552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:22.681552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.681552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:22.681900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.681900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:22.681900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:22.681900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:22.681900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.681900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:22.681900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:22.681900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:51:22.681900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:22.681900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:22.681900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.681900 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:22.682172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:22.682172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:22.682172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.682172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:22.682172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:22.682172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:22.682172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:22.682172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:22.682172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:22.682172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:22.682172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:22.682172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:22.682172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:22.682172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:22.682491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:22.682491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:22.682491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.682491 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:22.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:22.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:22.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:22.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:22.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:22.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:22.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:22.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:22.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:22.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:22.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:22.683055 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.683811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:22.683811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:22.683811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:51:22.683811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:22.683811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:22.683811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.683811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:22.683811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:22.683811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:22.683811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.683811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:22.683811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:22.683811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:22.683811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:22.683811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:22.683811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:22.683811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:22.683811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:22.684514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:22.684514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:22.684514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:22.684514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:22.684514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:22.684514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.684514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:22.684514 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:22.684514 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:22.684514 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:22.684721 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:22.684721 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:22.684721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.684887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:22.684887 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:22.684887 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:22.684887 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:22.684887 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:22.684887 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:22.684887 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.684887 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:22.685475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.685475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:22.685475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:22.685475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:22.685475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.685475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:22.685475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:22.685475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:22.685475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.685475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:22.685475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:22.685475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:22.685475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:22.685475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:22.685475 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:22.685754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:22.685754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:22.685754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:22.685754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:22.685754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:22.685754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:22.685754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:22.685754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.685754 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:22.686233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.686233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:22.686233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:22.686233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:22.686233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.686233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:22.686233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:22.686233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:22.686233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:22.686233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.686233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:22.686534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:22.686534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:22.686534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.686534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:22.686534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:22.686534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:22.686534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:22.686534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:22.686534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:22.686534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:22.686534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:22.686534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:22.686534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:22.686534 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:22.686898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:22.686898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:22.686898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.686898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:22.687002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.687002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:22.687002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:22.687002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:22.687002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:22.687002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:22.687002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.687002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:22.687002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:22.687002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.687002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:22.687002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:22.687002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.687002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:22.687002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:22.687002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.687002 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:22.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:22.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:22.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:22.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:22.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:22.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:22.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:22.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:22.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:22.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:22.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:22.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:22.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:22.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:22.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:22.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:22.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:22.687938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:22.687938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:22.687938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.687938 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:22.692436 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:22.692675 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:22.692767 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:22.692909 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:22.693169 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:22.693389 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:22.693526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.693929 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:22.693929 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:22.693929 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:22.693929 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:22.693929 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:22.693929 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:22.693929 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.693929 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:22.694567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.694567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:22.694567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:22.694567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:22.694567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.694567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:22.694567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:22.694567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.694567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:22.694567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:22.694567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:22.694567 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.694876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:22.694876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:22.694876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:22.694876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:22.694876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:22.694876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:22.694876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:22.694876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:22.694876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:22.694876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:22.694876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:22.694876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:22.694876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:22.695217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.695217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:22.695305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.695305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:22.695305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:22.695305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:22.695305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:22.695305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:22.695305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.695305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:22.695305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:22.695305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.695305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:22.695305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:22.695305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.695305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:22.695305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:22.695305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:22.695743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:22.695743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:22.695743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.695743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:22.695743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:22.695743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:22.695743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:22.695743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:22.695743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:22.695743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:22.695743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:22.695743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:22.695743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:22.695743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:22.695743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:22.695743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:22.695743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:22.695743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:22.695743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:22.696024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:22.696024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:22.696024 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:23.876496 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:23.876984 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:23.876984 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:23.877600 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:23.903936 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:23.904008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:23.904555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:23.904555 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:23.904555 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:23.904555 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:23.904555 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:23.904555 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:23.904555 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:23.904555 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:23.905240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:23.905240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:23.905240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:23.905240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:23.905240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:23.905240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:23.905240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:23.905240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:23.905240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:23.905240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:23.905240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:23.905240 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:23.905516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:23.905516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:23.905516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:23.905516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:23.905516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:23.905516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:23.905516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:23.905516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:23.905516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:23.905516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:23.905516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:23.905516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:23.905516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:23.905759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:23.905759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:23.905963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:23.905963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:23.905963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:23.905963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:23.905963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:23.905963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:23.905963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:23.905963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:23.905963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:23.905963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:23.905963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:23.905963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:23.905963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:23.905963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:23.905963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:23.905963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:23.906246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:23.906246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:23.906246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:23.906246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:23.906246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:23.906246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:23.906246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:23.906246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:23.906246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:23.906246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:23.906246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:23.906246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:23.906246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:23.906246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:23.906246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:23.906246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:23.906246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:23.906246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:23.906246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:23.906612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:23.906612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:23.906612 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:27.673863 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:27.674534 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:27.674534 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:27.675183 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:27.675433 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:27.675580 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:27.675756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.676393 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:27.676393 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:27.676393 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:27.676393 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:27.676393 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:27.676393 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:27.676393 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.676393 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:27.677244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.677244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:27.677244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:27.677244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:27.677244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.677244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:27.677244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:27.677244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.677244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:27.677244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:27.677244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:27.677244 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.677679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:27.677679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:27.677679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:27.677679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:27.677679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:27.677679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:27.677679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:27.677679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:27.677679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:27.677679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:27.677679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:27.677679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:27.677679 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:27.678041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.678041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:27.678041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.678041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:27.678041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:27.678041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:27.678041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:27.678041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:27.678041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.678041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:27.678041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:27.678041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.678041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:27.678041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:27.678041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.678512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:27.678512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:27.678512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.678512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:27.678512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:27.678512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.678512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:27.678512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:27.678512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:27.678512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.678512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:27.678512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:27.678512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:27.678512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:27.678512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:27.678512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:27.678512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:27.678512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:27.678512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:27.679122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:27.679122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:27.679122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:27.679122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:27.679122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.679122 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:27.679122 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:27.679122 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:27.679122 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:27.679122 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:51:27.679529 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:27.679755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.680277 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:27.680277 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:27.680277 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:27.680277 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:27.680277 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:27.680277 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:27.680277 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.680277 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:27.680851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.680851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:27.680851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:27.680851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:27.680851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.680851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:27.680851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:27.680851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:27.680851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.680851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:27.680851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:27.680851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:27.680851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:27.680851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:27.680851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:27.681289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:27.681289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:27.681289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:27.681289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:27.681289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:27.681289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:27.681289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:27.681289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.681289 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:27.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:27.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:27.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:27.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:27.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:27.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:51:27.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:27.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:27.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.681558 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:27.681928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:27.681928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:27.681928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.681928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:27.681928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:27.681928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:27.681928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:27.681928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:27.681928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:27.681928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:27.681928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:27.681928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:27.681928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:27.681928 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:27.682320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:27.682320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:27.682320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.682320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:27.682320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.682320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:27.682320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:27.682320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:27.682320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:27.682320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:27.682320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.682320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:27.682320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:27.682320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.682732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:27.682732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:27.682732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.682732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:27.682732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:27.682732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.682732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:27.682732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:27.682732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:51:27.682732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:27.682732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:27.682732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.682732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:27.682732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:27.682732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:27.682732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.682732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:27.682732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:27.682732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:27.683238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:27.683238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:27.683238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:27.683238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:27.683238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:27.683238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:27.683238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:27.683238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:27.683238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:27.683238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:27.683238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.683238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:27.683587 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:27.683848 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:27.683961 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:27.684073 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:27.684371 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:27.684563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.684937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:27.684937 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:27.684937 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:27.684937 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:27.684937 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:27.684937 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:27.684937 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.684937 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:27.685477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.685477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:27.685477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:27.685477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:27.685477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.685477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:27.685477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:27.685477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:27.685477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.685477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:27.685477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:27.685477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:27.685477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:27.685477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:27.685477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:27.685906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:27.685906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:27.685906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:27.685906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:27.685906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:27.685906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:27.685906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:27.685906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.685906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:27.686281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.686281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:27.686281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:27.686281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:27.686281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.686281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:27.686281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:27.686281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:27.686281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:27.686281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.686281 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:27.686608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:27.686608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:27.686608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.686608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:27.686608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:27.686608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:27.686608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:27.686608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:27.686608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:27.686608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:27.686608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:27.686608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:27.686608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:27.686608 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:27.687004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:27.687004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:27.687004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.687004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:27.687004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.687004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:27.687004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:27.687004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:27.687004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:27.687004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:27.687004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.687004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:27.687004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:27.687004 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:27.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:27.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:27.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:27.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:27.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:27.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:27.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:27.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:27.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:27.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:27.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:27.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:27.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:27.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:27.687905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:27.687905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:27.687905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:27.687905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:27.687905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:27.687905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:27.687905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:27.687905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:27.687905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:27.687905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.687905 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:27.692918 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:27.693201 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:27.693330 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:27.693490 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:27.693824 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:27.694060 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:27.694212 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.694731 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:27.694731 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:27.694731 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:27.694731 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:27.694731 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:27.694731 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:27.694731 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.694731 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:27.695336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.695336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:27.695336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:27.695336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:27.695336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.695336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:27.695336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:27.695336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.695336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:27.695336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:27.695336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:27.695336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.695681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:27.695681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:27.695681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:27.695681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:27.695681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:27.695681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:27.695681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:27.695681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:27.695681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:27.695681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:27.695681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:27.695681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:27.695681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:27.695989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.695989 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:27.696136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.696136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:27.696136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:27.696136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:27.696136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:27.696136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:27.696136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.696136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:27.696136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:27.696136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.696136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:27.696136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:27.696136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.696136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:27.696136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:27.696136 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:27.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:27.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:27.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:27.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:27.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:27.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:27.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:27.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:27.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:27.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:27.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:27.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:27.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:27.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:27.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:27.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:27.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:27.696575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:27.697049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:27.697049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:27.697049 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:28.876892 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:28.877494 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:28.877494 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:28.877820 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:28.914689 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:28.914986 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:28.915571 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:28.915571 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:28.915571 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:28.915571 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:28.915571 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:28.915571 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:28.915571 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:28.915571 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:28.916241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:28.916241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:28.916241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:28.916241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:28.916241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:28.916241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:28.916241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:28.916241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:28.916241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:28.916241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:28.916241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:28.916241 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:28.916538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:28.916538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:28.916538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:28.916538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:28.916538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:28.916538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:28.916538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:28.916538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:28.916538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:28.916538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:28.916538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:28.916538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:28.916538 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:28.916756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:28.916756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:28.917317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:28.917317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:28.917317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:28.917317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:28.917317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:28.917317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:28.917317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:28.917317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:28.917317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:28.917317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:28.917317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:28.917317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:28.917317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:28.917317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:28.917317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:28.917317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:28.917615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:28.917615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:28.917615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:28.917615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:28.917615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:28.917615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:28.917615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:28.917615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:28.917615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:28.917615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:28.917615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:28.917615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:28.917615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:28.917615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:28.917615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:28.917615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:28.917615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:28.917615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:28.917615 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:28.917911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:28.917911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:28.917911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:32.674423 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:32.675047 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:32.675047 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:32.675290 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:32.675427 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:32.675533 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:32.675699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.676103 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:32.676103 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:32.676103 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:32.676103 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:32.676103 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:32.676103 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:32.676103 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.676103 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:32.676983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.676983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:32.676983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:32.676983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:32.676983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.676983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:32.676983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:32.676983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.676983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:32.676983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:32.676983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:32.676983 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.677302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:32.677302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:32.677302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:32.677302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:32.677302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:32.677302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:32.677302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:32.677302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:32.677302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:32.677302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:32.677302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:32.677302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:32.677302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:32.677597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.677597 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:32.677677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.677677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:32.677677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:32.677677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:32.677677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:32.677677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:32.677677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.677677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:32.677677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:32.677677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.677677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:32.677677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:32.677677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.677677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:32.677677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:32.677677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.678021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:32.678021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:32.678021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.678021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:32.678021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:32.678021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:32.678021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.678021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:32.678021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:32.678021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:32.678021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:32.678021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:32.678021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:32.678021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:32.678021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:32.678021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:32.678021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:32.678021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:32.678021 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:32.678494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:32.678494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.678494 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:32.679008 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:32.679199 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:32.679320 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:32.679475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:51:32.679769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:32.679931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.680286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:32.680286 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:32.680286 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:32.680286 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:32.680286 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:32.680286 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:32.680286 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.680286 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:32.680803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.680803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:32.680803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:32.680803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:32.680803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.680803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:32.680803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:32.680803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:32.680803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.680803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:32.680803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:32.680803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:32.680803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:32.680803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:32.680803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:32.681245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:32.681245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:32.681245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:32.681245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:32.681245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:32.681245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:32.681245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:32.681245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.681245 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:32.681530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.681530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:32.681530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:32.681530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:32.681530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.681530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:32.681530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:32.681530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:51:32.681530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:32.681530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:32.681530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.681530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:32.681922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:32.681922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:32.681922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.681922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:32.681922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:32.681922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:32.681922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:32.681922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:32.681922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:32.681922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:32.681922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:32.681922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:32.681922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:32.681922 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:32.682311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:32.682311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:32.682311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.682311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:32.682311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.682311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:32.682311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:32.682311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:32.682311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:32.682311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:32.682311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.682311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:32.682311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:32.682311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.682695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:32.682695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:32.682695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.682695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:32.682695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:32.682695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.682695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:32.682695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:32.682695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:51:32.682695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:32.682695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:32.682695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.682695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:32.682695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:32.682695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:32.682695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.682695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:32.682695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:32.682695 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:32.683248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:32.683248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:32.683248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:32.683248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:32.683248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:32.683248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:32.683248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:32.683248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:32.683248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:32.683248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:32.683248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.683248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:32.683883 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:32.684037 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:32.684111 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:32.684223 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:32.684467 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:32.684712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.685013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:32.685013 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:32.685013 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:32.685013 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:32.685013 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:32.685013 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:32.685013 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.685013 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:32.685517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.685517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:32.685517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:32.685517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:32.685517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.685517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:32.685517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:32.685517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:32.685517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.685517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:32.685517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:32.685517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:32.685517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:32.685517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:32.685517 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:32.685947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:32.685947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:32.685947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:32.685947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:32.685947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:32.685947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:32.685947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:32.685947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.685947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:32.686222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.686222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:32.686222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:32.686222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:32.686222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.686222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:32.686222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:32.686222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:32.686222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:32.686222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.686222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:32.686585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:32.686585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:32.686585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.686585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:32.686585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:32.686585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:32.686585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:32.686585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:32.686585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:32.686585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:32.686585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:32.686585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:32.686585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:32.686585 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:32.687001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:32.687001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:32.687001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.687001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:32.687001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.687001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:32.687001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:32.687001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:32.687001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:32.687001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:32.687001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.687001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:32.687001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:32.687001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:32.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:32.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:32.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:32.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:32.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:32.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:32.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:32.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:32.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:32.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:32.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:32.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:32.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:32.687444 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:32.687956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:32.687956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:32.687956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:32.687956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:32.687956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:32.687956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:32.687956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:32.687956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:32.687956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:32.687956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.687956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:32.693239 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:32.693494 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:32.693600 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:32.693769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:32.694076 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:32.694269 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:32.694416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.694798 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:32.694798 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:32.694798 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:32.694798 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:32.694798 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:32.694798 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:32.694798 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.694798 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:32.695367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.695367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:32.695367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:32.695367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:32.695367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.695367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:32.695367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:32.695367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.695367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:32.695367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:32.695367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:32.695367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.695727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:32.695727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:32.695727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:32.695727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:32.695727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:32.695727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:32.695727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:32.695727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:32.695727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:32.695727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:32.695727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:32.695727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:32.695727 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:32.696085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.696085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:32.696085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.696085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:32.696085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:32.696085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:32.696085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:32.696085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:32.696085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.696085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:32.696085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:32.696085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.696085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:32.696085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:32.696085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.696544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:32.696544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:32.696544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:32.696544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:32.696544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:32.696544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.696544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:32.696544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:32.696544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:32.696544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:32.696544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:32.696544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:32.696544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:32.696544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:32.696544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:32.696544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:32.696544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:32.696544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:32.696544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:32.697157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:32.697157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:32.697157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:32.697157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:32.697157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:32.697157 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:33.878335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:33.878335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:33.878335 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:33.879063 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:33.914230 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:33.914530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:33.915222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:33.915222 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:33.915222 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:33.915222 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:33.915222 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:33.915222 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:33.915222 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:33.915222 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:33.917076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:33.917076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:33.917076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:33.917076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:33.917076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:33.917076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:33.917076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:33.917076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:33.917076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:33.917076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:33.917076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:33.917076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:33.917379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:33.917379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:33.917379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:33.917379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:33.917379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:33.917379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:33.917379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:33.917379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:33.917379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:33.917379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:33.917379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:33.917379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:33.917379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:33.917568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:33.917568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:33.917568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:33.917568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:33.917568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:33.917568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:33.917568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:33.917568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:33.917568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:33.917568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:33.917568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:33.917568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:33.917568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:33.917568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:33.917568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:33.917849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:33.917849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:33.917849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:33.917849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:33.917849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:33.917849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:33.917849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:33.917849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:33.917849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:33.917849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:33.917849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:33.917849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:33.917849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:33.917849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:33.917849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:33.917849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:33.917849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:33.917849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:33.917849 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:33.918201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:33.918201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:33.918201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:33.918201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:33.918201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:33.918201 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:37.674904 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:37.675360 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:37.675360 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:37.676017 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:37.676248 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:37.676425 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:37.676551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.677043 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:37.677043 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:37.677043 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:37.677043 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:37.677043 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:37.677043 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:37.677043 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.677043 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:37.677797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.677797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:37.677797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:37.677797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:37.677797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.677797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:37.677797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:37.677797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.677797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:37.677797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:37.677797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:37.677797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.678092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:37.678092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:37.678092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:37.678092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:37.678092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:37.678092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:37.678092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:37.678092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:37.678092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:37.678092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:37.678092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:37.678092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:37.678092 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:37.678371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.678371 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:37.678530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.678530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:37.678530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:37.678530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:37.678530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:37.678530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:37.678530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.678530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:37.678530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:37.678530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.678530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:37.678530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:37.678530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.678530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:37.678530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:37.678530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.678812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:37.678812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:37.678812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.678812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:37.678812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:37.678812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:37.678812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.678812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:37.678812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:37.678812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:37.678812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:37.678812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:37.678812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:37.678812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:37.678812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:37.678812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:37.678812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:37.678812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:37.678812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:37.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:37.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.679111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:37.679234 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:37.679653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:37.679801 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:37.679945 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:51:37.680285 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:37.680551 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.681015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:37.681015 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:37.681015 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:37.681015 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:37.681015 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:37.681015 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:37.681015 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.681015 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:37.681572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.681572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:37.681572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:37.681572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:37.681572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.681572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:37.681572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:37.681572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:37.681572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.681572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:37.681572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:37.681572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:37.681572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:37.681572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:37.681572 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:37.682028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:37.682028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:37.682028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:37.682028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:37.682028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:37.682028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:37.682028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:37.682028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.682028 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:37.682264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.682264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:37.682264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:37.682264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:37.682264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.682264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:37.682264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:37.682264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:51:37.682264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:37.682264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:37.682264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.682264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:37.682521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:37.682521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:37.682521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.682521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:37.682521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:37.682521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:37.682521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:37.682521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:37.682521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:37.682521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:37.682521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:37.682521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:37.682521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:37.682521 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:37.682746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:37.682746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:37.682746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.682746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:37.682816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.682816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:37.682816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:37.682816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:37.682816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:37.682816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:37.682816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.682816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:37.682816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:37.682816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.682816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:37.682816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:37.682816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.682816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:37.682816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:37.682816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.683058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:37.683058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:37.683058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:51:37.683058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:37.683058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:37.683058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.683058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:37.683058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:37.683058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:37.683058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.683058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:37.683058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:37.683058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:37.683058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:37.683058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:37.683058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:37.683058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:37.683058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:37.683327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:37.683327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:37.683327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:37.683327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:37.683327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:37.683327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.683327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:37.684099 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:37.684265 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:37.684376 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:37.684468 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:37.684972 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:37.685301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.685902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:37.685902 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:37.685902 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:37.685902 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:37.685902 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:37.685902 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:37.685902 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.685902 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:37.686394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.686394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:37.686394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:37.686394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:37.686394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.686394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:37.686394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:37.686394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:37.686394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.686394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:37.686394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:37.686394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:37.686394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:37.686394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:37.686394 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:37.686656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:37.686656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:37.686656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:37.686656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:37.686656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:37.686656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:37.686656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:37.686656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.686656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:37.687095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.687095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:37.687095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:37.687095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:37.687095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.687095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:37.687095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:37.687095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:37.687095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:37.687095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.687095 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:37.687329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:37.687329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:37.687329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.687329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:37.687329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:37.687329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:37.687329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:37.687329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:37.687329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:37.687329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:37.687329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:37.687329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:37.687329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:37.687329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:37.687604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:37.687604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:37.687604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.687604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:37.688235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.688235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:37.688235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:37.688235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:37.688235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:37.688235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:37.688235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.688235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:37.688235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:37.688235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.688235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:37.688235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:37.688235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.688235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:37.688235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:37.688235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.688235 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:37.688577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:37.688577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:37.688577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:37.688577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.688577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:37.688577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:37.688577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:37.688577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.688577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:37.688577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:37.688577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:37.688577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:37.688577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:37.688577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:37.688577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:37.688577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:37.688577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:37.688577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:37.688577 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:37.688873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:37.688873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:37.688873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.688873 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:37.693559 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:37.693796 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:37.693934 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:37.694117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:37.694589 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:37.694886 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:37.695080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.695599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:37.695599 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:37.695599 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:37.695599 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:37.695599 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:37.695599 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:37.695599 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.695599 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:37.696601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.696601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:37.696601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:37.696601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:37.696601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.696601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:37.696601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:37.696601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.696601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:37.696601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:37.696601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:37.696601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.697016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:37.697016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:37.697016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:37.697016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:37.697016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:37.697016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:37.697016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:37.697016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:37.697016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:37.697016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:37.697016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:37.697016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:37.697016 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:37.697228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.697228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:37.697546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.697546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:37.697546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:37.697546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:37.697546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:37.697546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:37.697546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.697546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:37.697546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:37.697546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.697546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:37.697546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:37.697546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.697546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:37.697546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:37.697546 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:37.697881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:37.697881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:37.697881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.697881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:37.697881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:37.697881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:37.697881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:37.697881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:37.697881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:37.697881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:37.697881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:37.697881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:37.697881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:37.697881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:37.697881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:37.697881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:37.697881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:37.697881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:37.697881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:37.698272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:37.698272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:37.698272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:38.877840 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:38.878404 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:38.878404 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:38.879222 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:38.912989 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:38.913183 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:38.913762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:38.913762 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:38.913762 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:38.913762 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:38.913762 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:38.913762 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:38.913762 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:38.913762 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:38.914761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:38.914761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:38.914761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:38.914761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:38.914761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:38.914761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:38.914761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:38.914761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:38.914761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:38.914761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:38.914761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:38.914761 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:38.915118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:38.915118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:38.915118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:38.915118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:38.915118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:38.915118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:38.915118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:38.915118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:38.915118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:38.915118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:38.915118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:38.915118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:38.915118 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:38.915438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:38.915438 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:38.915814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:38.915814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:38.915814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:38.915814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:38.915814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:38.915814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:38.915814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:38.915814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:38.915814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:38.915814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:38.915814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:38.915814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:38.915814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:38.915814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:38.915814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:38.915814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:38.916346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:38.916346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:38.916346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:38.916346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:38.916346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:38.916346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:38.916346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:38.916346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:38.916346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:38.916346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:38.916346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:38.916346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:38.916346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:38.916346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:38.916346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:38.916346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:38.916346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:38.916346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:38.916346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:38.916853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:38.916853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:38.916853 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:42.675224 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:42.675768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:42.675768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:42.676648 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:42.676851 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-77ab8ddd-dee1-41ae-b38b-83c0f140f13d, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:42.676980 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1e6d0d4f-769b-40ee-9f46-dc2dd219205c, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:42.677096 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.677559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:42.677559 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:42.677559 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:42.677559 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:42.677559 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:42.677559 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:42.677559 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.677559 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:42.678233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.678233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:42.678233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:42.678233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:42.678233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.678233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:42.678233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:42.678233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.678233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:42.678233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:42.678233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:42.678233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.678691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:42.678691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:42.678691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:42.678691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:42.678691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:42.678691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:42.678691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:42.678691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:42.678691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:42.678691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:42.678691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:42.678691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:42.678691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:42.679087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.679087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:42.679087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.679087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:42.679087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:42.679087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:42.679087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:42.679087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:42.679087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.679087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:42.679087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:42.679087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.679087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:42.679087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:42.679087 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.679387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:42.679387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:42.679387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.679387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:42.679387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:42.679387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.679387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:42.679387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:42.679387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:42.679387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.679387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:42.679387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:42.679387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:42.679387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:42.679387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:42.679387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:42.679387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:42.679387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:42.679387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:42.679706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:42.679706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:42.679706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:42.679706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:42.679706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.679706 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:42.679706 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:42.679706 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:42.679851 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:42.679851 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:51:42.680137 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:42.680305 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.680697 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:42.680697 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:42.680697 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:42.680697 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:42.680697 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:42.680697 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:42.680697 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.680697 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:42.681285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.681285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:42.681285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:42.681285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:42.681285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.681285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:42.681285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:42.681285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:42.681285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.681285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:42.681285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:42.681285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:42.681285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:42.681285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:42.681285 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:42.681851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:42.681851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:42.681851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:42.681851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:42.681851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:42.681851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:42.681851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:42.681851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.681851 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:42.682076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.682076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:42.682076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:42.682076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:42.682076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.682076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:42.682076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:42.682076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:51:42.682076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:42.682076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:42.682076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.682076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:42.682376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:42.682376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:42.682376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.682376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:42.682376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:42.682376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:42.682376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:42.682376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:42.682376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:42.682376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:42.682376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:42.682376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:42.682376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:42.682376 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:42.682826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:42.682826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:42.682826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.682826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:42.682826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.682826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:42.682826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:42.682826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:42.682826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:42.682826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:42.682826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.682826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:42.682826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:42.682826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.683214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:42.683214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:42.683214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.683214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:42.683214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:42.683214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.683214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:42.683214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:42.683214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:51:42.683214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:42.683214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:42.683214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.683214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:42.683214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:42.683214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:42.683214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.683214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:42.683214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:42.683214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:42.683678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:42.683678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:42.683678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:42.683678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:42.683678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:42.683678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:42.683678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:42.683678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:42.683678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:42.683678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:42.683678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.683678 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:42.684510 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:42.684510 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:42.684653 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:42.684696 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:42.685033 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:42.685218 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.685658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:42.685658 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:42.685658 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:42.685658 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:42.685658 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:42.685658 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:42.685658 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.685658 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:42.686152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.686152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:42.686152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:42.686152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:42.686152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.686152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:42.686152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:42.686152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:42.686152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.686152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:42.686152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:42.686152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:42.686152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:42.686152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:42.686152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:42.686557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:42.686557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:42.686557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:42.686557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:42.686557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:42.686557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:42.686557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:42.686557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.686557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:42.686785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.686785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:42.686785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:42.686785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:42.686785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.686785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:42.686785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:42.686785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:42.686785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:42.686785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.686785 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:42.687077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:42.687077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:42.687077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.687077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:42.687077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:42.687077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:42.687077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:42.687077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:42.687077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:42.687077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:42.687077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:42.687077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:42.687077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:42.687077 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:42.687430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:42.687430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:42.687430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.687430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:42.687430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.687430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:42.687430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:42.687430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:42.687430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:42.687430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:42.687430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.687430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:42.687430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:42.687430 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.687803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:42.687803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:42.687803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.687803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:42.687803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:42.687803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.687803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:42.687803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:42.687803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:42.687803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:42.687803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.687803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:42.687803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:42.687803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:42.687803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.687803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:42.687803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:42.687803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:42.687803 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:42.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:42.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:42.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:42.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:42.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:42.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:42.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:42.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:42.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:42.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.688246 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:42.693914 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:42.694323 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:42.694477 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:42.694744 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:42.695134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:42.695424 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:42.695559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.696019 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:42.696019 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:42.696019 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:42.696019 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:42.696019 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:42.696019 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:42.696019 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.696019 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:42.696861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.696861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:42.696861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:42.696861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:42.696861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.696861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:42.696861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:42.696861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.696861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:42.696861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:42.696861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:42.696861 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.697097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:42.697097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:42.697097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:42.697097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:42.697097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:42.697097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:42.697097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:42.697097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:42.697097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:42.697097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:42.697097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:42.697097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:42.697097 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:42.697349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.697349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:42.697806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.697806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:42.697806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:42.697806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:42.697806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:42.697806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:42.697806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.697806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:42.697806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:42.697806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.697806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:42.697806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:42.697806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.697806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:42.697806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:42.697806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:42.698112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:42.698112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:42.698112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.698112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:42.698112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:42.698112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:42.698112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:42.698112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:42.698112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:42.698112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:42.698112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:42.698112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:42.698112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:42.698112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:42.698112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:42.698112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:42.698112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:42.698112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:42.698112 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:42.698439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:42.698439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:42.698439 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:43.878220 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:43.878701 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:43.878802 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:43.879381 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:43.903443 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:43.903742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:43.904264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:43.904264 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:43.904264 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:43.904264 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:43.904264 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:43.904264 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:43.904264 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:43.904264 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:43.904968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:43.904968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:43.904968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:43.904968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:43.904968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:43.904968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:43.904968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:43.904968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:43.904968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:43.904968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:43.904968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:43.904968 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:43.905237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:43.905237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:43.905237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:43.905237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:43.905237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:43.905237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:43.905237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:43.905237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:43.905237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:43.905237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:43.905237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:43.905237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:43.905237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:43.905480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:43.905480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:43.905696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:43.905696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:43.905696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:43.905696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:43.905696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:43.905696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:43.905696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:43.905696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:43.905696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:43.905696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:43.905696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:43.905696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:43.905696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:43.905696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:43.905696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:43.905696 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:43.905974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:43.905974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:43.905974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:43.905974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:43.905974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:43.905974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:43.905974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:43.905974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:43.905974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:43.905974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:43.905974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:43.905974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:43.905974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:43.905974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:43.905974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:43.905974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:43.905974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:43.905974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:43.905974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:43.906296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:43.906296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:43.906296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:47.675851 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:47.676347 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:47.676347 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:47.677031 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:47.677248 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.677823 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:47.677823 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:47.677823 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:47.677823 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:47.677823 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:47.677823 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:47.677823 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.677823 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:47.678903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.678903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:47.678903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:47.678903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:47.678903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.678903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:47.678903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:47.678903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.678903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:47.678903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:47.678903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:47.678903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.679329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:47.679329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:47.679329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:47.679329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:47.679329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:47.679329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:47.679329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:47.679329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:47.679329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:47.679329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:47.679329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:47.679329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:47.679329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:47.679717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.679717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:47.679893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.679893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:47.679893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:47.679893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:47.679893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:47.679893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:47.679893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.679893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:47.679893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:47.679893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.679893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:47.679893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:47.679893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.679893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:47.679893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:47.679893 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.680206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:47.680206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:47.680206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.680206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:47.680206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:47.680206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:47.680206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.680206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:47.680206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:47.680206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:47.680206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:47.680206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:47.680206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:47.680206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:47.680206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:47.680206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:47.680206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:47.680206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:47.680206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:47.680565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:47.680565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.680565 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:47.680565 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:47.680565 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:47.680565 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:47.680565 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:51:47.680883 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:47.681040 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.681370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:47.681370 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:47.681370 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:47.681370 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:47.681370 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:47.681370 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:47.681370 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.681370 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:47.681862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.681862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:47.681862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:47.681862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:47.681862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.681862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:47.681862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:47.681862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:47.681862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.681862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:47.681862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:47.681862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:47.681862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:47.681862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:47.681862 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:47.682130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:47.682130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:47.682130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:47.682130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:47.682130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:47.682130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:47.682130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:47.682130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.682130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:47.682839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.682839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:47.682839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:47.682839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:47.682839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.682839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:47.682839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:47.682839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:51:47.682839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:47.682839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:47.682839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.682839 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:47.683098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:47.683098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:47.683098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.683098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:47.683098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:47.683098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:47.683098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:47.683098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:47.683098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:47.683098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:47.683098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:47.683098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:47.683098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:47.683098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:47.683358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:47.683358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:47.683358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.683358 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:47.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:47.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:47.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:47.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:47.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:47.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:47.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:47.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:47.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:47.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:47.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:47.683620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.684042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:47.684042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:47.684042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:51:47.684042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:47.684042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:47.684042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.684042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:47.684042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:47.684042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:47.684042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.684042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:47.684042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:47.684042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:47.684042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:47.684042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:47.684042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:47.684042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:47.684042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:47.684632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:47.684632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:47.684632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:47.684632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:47.684632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:47.684632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.684632 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:47.684632 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:47.684855 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:47.684936 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:47.685079 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:47.685367 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:47.685544 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.685892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:47.685892 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:47.685892 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:47.685892 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:47.685892 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:47.685892 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:47.685892 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.685892 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:47.686541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.686541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:47.686541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:47.686541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:47.686541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.686541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:47.686541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:47.686541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:47.686541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.686541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:47.686541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:47.686541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:47.686541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:47.686541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:47.686541 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:47.686869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:47.686869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:47.686869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:47.686869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:47.686869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:47.686869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:47.686869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:47.686869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.686869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:47.687115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.687115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:47.687115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:47.687115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:47.687115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.687115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:47.687115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:47.687115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:47.687115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:47.687115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.687115 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:47.687314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:47.687314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:47.687314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.687314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:47.687314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:47.687314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:47.687314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:47.687314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:47.687314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:47.687314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:47.687314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:47.687314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:47.687314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:47.687314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:47.687596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:47.687596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:47.687596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.687596 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:47.688084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.688084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:47.688084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:47.688084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:47.688084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:47.688084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:47.688084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.688084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:47.688084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:47.688084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.688084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:47.688084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:47.688084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.688084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:47.688084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:47.688084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.688084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:47.688354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:47.688354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:47.688354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:47.688354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.688354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:47.688354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:47.688354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:47.688354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.688354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:47.688354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:47.688354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:47.688354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:47.688354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:47.688354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:47.688354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:47.688354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:47.688354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:47.688354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:47.688354 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:47.688669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:47.688669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:47.688669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.688669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:47.694337 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:47.694631 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:47.694705 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:47.694867 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:47.695364 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:47.695650 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:47.695772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.696311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:47.696311 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:47.696311 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:47.696311 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:47.696311 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:47.696311 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:47.696311 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.696311 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:47.697317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.697317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:47.697317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:47.697317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:47.697317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.697317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:47.697317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:47.697317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.697317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:47.697317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:47.697317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:47.697317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.697616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:47.697616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:47.697616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:47.697616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:47.697616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:47.697616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:47.697616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:47.697616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:47.697616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:47.697616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:47.697616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:47.697616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:47.697616 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:47.697936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.697936 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:47.698408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.698408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:47.698408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:47.698408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:47.698408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:47.698408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:47.698408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.698408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:47.698408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:47.698408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.698408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:47.698408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:47.698408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.698408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:47.698408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:47.698408 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:47.698820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:47.698820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:47.698820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.698820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:47.698820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:47.698820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:47.698820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:47.698820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:47.698820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:47.698820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:47.698820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:47.698820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:47.698820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:47.698820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:47.698820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:47.698820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:47.698820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:47.698820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:47.698820 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:47.699155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:47.699155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:47.699155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:48.878781 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:48.879366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:48.879366 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:48.879953 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:48.910442 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:48.910767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:48.911485 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:48.911485 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:48.911485 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:48.911485 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:48.911485 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:48.911485 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:48.911485 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:48.911485 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:48.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:48.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:48.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:48.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:48.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:48.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:48.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:48.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:48.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:48.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:48.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:48.912298 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:48.912530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:48.912530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:48.912530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:48.912530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:48.912530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:48.912530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:48.912530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:48.912530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:48.912530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:48.912530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:48.912530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:48.912530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:48.912530 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:48.912784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:48.912784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:48.913057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:48.913057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:48.913057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:48.913057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:48.913057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:48.913057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:48.913057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:48.913057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:48.913057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:48.913057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:48.913057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:48.913057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:48.913057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:48.913057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:48.913057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:48.913057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:48.913387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:48.913387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:48.913387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:48.913387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:48.913387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:48.913387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:48.913387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:48.913387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:48.913387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:48.913387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:48.913387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:48.913387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:48.913387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:48.913387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:48.913387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:48.913387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:48.913387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:48.913387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:48.913387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:48.913768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:48.913768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:48.913768 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:52.676333 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:52.677176 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:52.677176 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:52.677669 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:52.677898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.678495 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:52.678495 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:52.678495 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:52.678495 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:52.678495 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:52.678495 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:52.678495 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.678495 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:52.679200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.679200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:52.679200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:52.679200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:52.679200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.679200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:52.679200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:52.679200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.679200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:52.679200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:52.679200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:52.679200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.679642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:52.679642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:52.679642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:52.679642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:52.679642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:52.679642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:52.679642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:52.679642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:52.679642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:52.679642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:52.679642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:52.679642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:52.679642 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:52.680027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.680027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:52.680027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.680027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:52.680027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:52.680027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:52.680027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:52.680027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:52.680027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.680027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:52.680027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:52.680027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.680027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:52.680027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:52.680027 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.680531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:52.680531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:52.680531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.680531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:52.680531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:52.680531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.680531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:52.680531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:52.680531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:52.680531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.680531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:52.680531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:52.680531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:52.680531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:52.680531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:52.680531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:52.680531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:52.680531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:52.680531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:52.681058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:52.681058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:52.681058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:52.681058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:52.681058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.681058 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:52.681058 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:52.681058 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:52.681058 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:52.681058 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:51:52.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:52.681351 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.681759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:52.681759 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:52.681759 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:52.681759 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:52.681759 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:52.681759 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:52.681759 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.681759 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:52.682280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.682280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:52.682280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:52.682280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:52.682280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.682280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:52.682280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:52.682280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:52.682280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.682280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:52.682280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:52.682280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:52.682280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:52.682280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:52.682280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:52.682709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:52.682709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:52.682709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:52.682709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:52.682709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:52.682709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:52.682709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:52.682709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.682709 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:52.683001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.683001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:52.683001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:52.683001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:52.683001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.683001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:52.683001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:52.683001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:51:52.683001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:52.683001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:52.683001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.683001 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:52.683349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:52.683349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:52.683349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.683349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:52.683349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:52.683349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:52.683349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:52.683349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:52.683349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:52.683349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:52.683349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:52.683349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:52.683349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:52.683349 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:52.683680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:52.683680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:52.683680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.683680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:52.683680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.683680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:52.683680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:52.683680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:52.683680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:52.683680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:52.683680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.683680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:52.683680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:52.683680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:52.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:52.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:52.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:52.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:52.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:52.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:51:52.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:52.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:52.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:52.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:52.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:52.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:52.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:52.684064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:52.684850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:52.684850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:52.684850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:52.684850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:52.684850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:52.684850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:52.684850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:52.684850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:52.684850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:52.684850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:52.684850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.684850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:52.687700 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:52.687922 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:52.688028 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:52.688117 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:52.688507 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:52.688793 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.689272 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:52.689272 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:52.689272 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:52.689272 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:52.689272 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:52.689272 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:52.689272 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.689272 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:52.689817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.689817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:52.689817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:52.689817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:52.689817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.689817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:52.689817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:52.689817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:52.689817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.689817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:52.689817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:52.689817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:52.689817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:52.689817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:52.689817 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:52.690211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:52.690211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:52.690211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:52.690211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:52.690211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:52.690211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:52.690211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:52.690211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.690211 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:52.690471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.690471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:52.690471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:52.690471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:52.690471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.690471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:52.690471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:52.690471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:52.690471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:52.690471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.690471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:52.690764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:52.690764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:52.690764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.690764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:52.690764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:52.690764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:52.690764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:52.690764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:52.690764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:52.690764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:52.690764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:52.690764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:52.690764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:52.690764 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:52.691099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:52.691099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:52.691099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.691099 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:52.691205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.691205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:52.691205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:52.691205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:52.691205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:52.691205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:52.691205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.691205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:52.691205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:52.691205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.691205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:52.691205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:52.691205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.691205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:52.691205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:52.691205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.691205 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:52.691631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:52.691631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:52.691631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:52.691631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.691631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:52.691631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:52.691631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:52.691631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.691631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:52.691631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:52.691631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:52.691631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:52.691631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:52.691631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:52.691631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:52.691631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:52.691631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:52.691631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:52.691631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:52.692065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:52.692065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:52.692065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.692065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:52.694735 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:52.694963 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:52.695062 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:52.695203 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:52.695551 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:52.695815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:52.695943 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:52.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:52.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:52.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:52.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:52.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:52.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:52.697067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.697067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:52.697067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:52.697067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:52.697067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.697067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:52.697067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:52.697067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.697067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:52.697067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:52.697067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:52.697067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.697410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:52.697410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:52.697410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:52.697410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:52.697410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:52.697410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:52.697410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:52.697410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:52.697410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:52.697410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:52.697410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:52.697410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:52.697410 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:52.697742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.697742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:52.697742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.697742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:52.697742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:52.697742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:52.697742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:52.697742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:52.697742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.697742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:52.697742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:52.697742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.697742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:52.697742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:52.697742 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.698073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:52.698073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:52.698073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:52.698073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:52.698073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:52.698073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.698073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:52.698073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:52.698073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:52.698073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:52.698073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:52.698073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:52.698073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:52.698073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:52.698073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:52.698073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:52.698073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:52.698073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:52.698073 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:52.698526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:52.698526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:52.698526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:52.698526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:52.698526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:52.698526 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:53.879207 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:53.879734 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:53.879770 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:53.880543 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:53.886421 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=564,neutron:UPDATE=1 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:51:53.912666 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:53.912808 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:53.913239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:53.913239 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:53.913239 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:53.913239 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:53.913239 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:53.913239 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:53.913239 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:53.913239 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:53.914173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:53.914173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:53.914173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:53.914173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:53.914173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:53.914173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:53.914173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:53.914173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:53.914173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:53.914173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:53.914173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:53.914173 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:53.914479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:53.914479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:53.914479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:53.914479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:53.914479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:53.914479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:53.914479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:53.914479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:53.914479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:53.914479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:53.914479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:53.914479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:53.914479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:53.914688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:53.914688 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:53.915340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:53.915340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:53.915340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:53.915340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:53.915340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:53.915340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:53.915340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:53.915340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:53.915340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:53.915340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:53.915340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:53.915340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:53.915340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:53.915340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:53.915340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:53.915340 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:53.915671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:53.915671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:53.915671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:53.915671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:53.915671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:53.915671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:53.915671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:53.915671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:53.915671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:53.915671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:53.915671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:53.915671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:53.915671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:53.915671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:53.915671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:53.915671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:53.915671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:53.915671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:53.915671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:53.915964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:53.915964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:53.915964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:57.677010 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:57.677519 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:57.677710 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:57.678295 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:57.678484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.679039 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:57.679039 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:57.679039 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:57.679039 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:57.679039 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:57.679039 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:57.679039 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.679039 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:57.680069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.680069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:57.680069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:57.680069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:57.680069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.680069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:57.680069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:57.680069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.680069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:57.680069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:57.680069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:57.680069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.680429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:57.680429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:57.680429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:57.680429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:57.680429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:57.680429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:57.680429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:57.680429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:57.680429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:57.680429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:57.680429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:57.680429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:57.680429 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:57.680654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.680654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:57.681234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.681234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:57.681234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:57.681234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:57.681234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:57.681234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:57.681234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.681234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:57.681234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:57.681234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.681234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:57.681234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:57.681234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.681234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:57.681234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:57.681234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.681537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:51:57.681537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:57.681537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.681537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:57.681537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:57.681537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:57.681537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.681537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:57.681537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:57.681537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:57.681537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:57.681537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:57.681537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:57.681537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:57.681537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:57.681537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:57.681537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:57.681537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:57.681537 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:57.681838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:57.681838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.681838 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:57.681838 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:57.681910 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:57.681993 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:57.682123 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:51:57.682449 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:57.682604 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.683036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:57.683036 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:57.683036 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:57.683036 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:57.683036 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:57.683036 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:57.683036 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.683036 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:57.683821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.683821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:57.683821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:57.683821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:57.683821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.683821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:57.683821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:57.683821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:57.683821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.683821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:57.683821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:57.683821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:57.683821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:57.683821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:57.683821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:57.684228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:57.684228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:57.684228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:57.684228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:57.684228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:57.684228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:57.684228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:57.684228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.684228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:57.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:57.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:57.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:57.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:57.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:57.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:51:57.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:57.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:57.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:57.684999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:57.684999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:57.684999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.684999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:57.684999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:57.684999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:57.684999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:57.684999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:57.684999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:57.684999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:57.684999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:57.684999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:57.684999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:57.684999 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:57.685341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:57.685341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:57.685341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.685341 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:57.685720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.685720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:57.685720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:57.685720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:57.685720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:57.685720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:57.685720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.685720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:57.685720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:57.685720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.685720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:57.685720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:57.685720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.685720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:57.685720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:57.685720 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.686125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:51:57.686125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:51:57.686125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:51:57.686125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:57.686125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:57.686125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.686125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:57.686125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:57.686125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:57.686125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.686125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:57.686125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:57.686125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:57.686125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:57.686125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:57.686125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:57.686125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:57.686125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:57.686602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:57.686602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:57.686602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:57.686602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:57.686602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:57.686602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.686602 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:57.688203 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:57.688539 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:57.688670 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:57.688802 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:57.689106 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:57.689306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.689668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:57.689668 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:57.689668 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:57.689668 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:57.689668 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:57.689668 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:57.689668 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.689668 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:57.690411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.690411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:51:57.690411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:57.690411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:57.690411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.690411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:57.690411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:57.690411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:51:57.690411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.690411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:51:57.690411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:57.690411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:51:57.690411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:57.690411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:51:57.690411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:51:57.690795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:57.690795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:51:57.690795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:57.690795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:51:57.690795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:51:57.690795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:57.690795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:51:57.690795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.690795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:51:57.691317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.691317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:57.691317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:57.691317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:57.691317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.691317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:57.691317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:57.691317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:57.691317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:57.691317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.691317 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:57.691587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:57.691587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:57.691587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.691587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:57.691587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:57.691587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:57.691587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:57.691587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:57.691587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:57.691587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:57.691587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:57.691587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:57.691587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:57.691587 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:57.691944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:57.691944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:57.691944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.691944 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:57.692403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.692403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:57.692403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:57.692403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:57.692403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:57.692403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:57.692403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.692403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:57.692403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:57.692403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.692403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:57.692403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:57.692403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.692403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:57.692403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:57.692403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.692403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:51:57.692906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:51:57.692906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:51:57.692906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:51:57.692906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.692906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:57.692906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:57.692906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:57.692906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.692906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:57.692906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:57.692906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:57.692906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:57.692906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:57.692906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:57.692906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:57.692906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:57.692906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:57.692906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:57.692906 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:57.693364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:57.693364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:57.693364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.693364 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:57.695179 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:57.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:57.695645 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:57.695817 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:57.696122 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:57.696359 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:57.696456 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.696859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:57.696859 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:57.696859 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:57.696859 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:57.696859 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:57.696859 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:57.696859 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.696859 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:57.697733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.697733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:57.697733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:57.697733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:57.697733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.697733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:57.697733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:57.697733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.697733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:57.697733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:57.697733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:57.697733 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.698013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:57.698013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:57.698013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:57.698013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:57.698013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:57.698013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:57.698013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:57.698013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:57.698013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:57.698013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:57.698013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:57.698013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:57.698013 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:57.698329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.698329 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:57.698762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.698762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:57.698762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:57.698762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:57.698762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:57.698762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:57.698762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.698762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:57.698762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:57.698762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.698762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:57.698762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:57.698762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.698762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:57.698762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:57.698762 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:57.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:51:57.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:57.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:57.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:57.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:57.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:57.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:57.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:57.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:57.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:57.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:57.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:57.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:57.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:57.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:57.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:57.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:57.699525 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:57.700200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:57.700200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:57.700200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:51:58.879739 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:51:58.880285 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:51:58.880285 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:51:58.880975 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:51:58.909441 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:51:58.909670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:58.910214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:51:58.910214 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:58.910214 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:51:58.910214 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:51:58.910214 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:58.910214 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:51:58.910214 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:58.910214 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:51:58.910913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:58.910913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:51:58.910913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:58.910913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:51:58.910913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:58.910913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:58.910913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:58.910913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:58.910913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:58.910913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:58.910913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:51:58.910913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:58.911228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:51:58.911228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:58.911228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:51:58.911228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:58.911228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:51:58.911228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:51:58.911228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:58.911228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:51:58.911228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:58.911228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:51:58.911228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:51:58.911228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:58.911228 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:51:58.911489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:58.911489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:51:58.911766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:58.911766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:51:58.911766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:51:58.911766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:51:58.911766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:51:58.911766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:51:58.911766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:58.911766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:51:58.911766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:51:58.911766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:58.911766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:51:58.911766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:51:58.911766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:51:58.911766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:51:58.911766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:51:58.911766 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:51:58.912309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:51:58.912309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:51:58.912309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:58.912309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:58.912309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:51:58.912309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:51:58.912309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:51:58.912309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:51:58.912309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:51:58.912309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:51:58.912309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:51:58.912309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:51:58.912309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:51:58.912309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:51:58.912309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:51:58.912309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:51:58.912309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:51:58.912309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:51:58.912309 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:51:58.912830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:51:58.912830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:51:58.912830 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:02.677564 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:02.678081 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:02.678081 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:02.678542 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:02.678783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.679323 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:02.679323 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:02.679323 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:02.679323 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:02.679323 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:02.679323 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:02.679323 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.679323 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:02.680042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.680042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:02.680042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:02.680042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:02.680042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.680042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:02.680042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:02.680042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.680042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:02.680042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:02.680042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:02.680042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.680319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:02.680319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:02.680319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:02.680319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:02.680319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:02.680319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:02.680319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:02.680319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:02.680319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:02.680319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:02.680319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:02.680319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:02.680319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:02.680528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.680528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:02.680843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.680843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:02.680843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:02.680843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:02.680843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:02.680843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:02.680843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.680843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:02.680843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:02.680843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.680843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:02.680843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:02.680843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.680843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:02.680843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:02.680843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.681478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:02.681478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:02.681478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.681478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:02.681478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:02.681478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:02.681478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.681478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:02.681478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:02.681478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:02.681478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:02.681478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:02.681478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:02.681478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:02.681478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:02.681478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:02.681478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:02.681478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:02.681478 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:02.681816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:02.681816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.681816 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:02.681872 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:02.682104 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:02.682157 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:02.682237 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:52:02.682683 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:02.682898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.683378 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:02.683378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:02.683378 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:02.683378 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:02.683378 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:02.683378 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:02.683378 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.683378 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:02.683969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.683969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:02.683969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:02.683969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:02.683969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.683969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:02.683969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:02.683969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:02.683969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.683969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:02.683969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:02.683969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:02.683969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:02.683969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:02.683969 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:02.684515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:02.684515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:02.684515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:02.684515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:02.684515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:02.684515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:02.684515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:02.684515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.684515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:02.684677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.684677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:02.684677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:02.684677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:02.684677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.684677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:02.684677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:02.684677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:52:02.684677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:02.684677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:02.684677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.684677 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:02.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:02.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:02.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:02.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:02.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:02.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:02.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:02.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:02.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:02.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:02.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:02.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:02.684960 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:02.685165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:02.685165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:02.685165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.685165 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:02.685480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.685480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:02.685480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:02.685480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:02.685480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:02.685480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:02.685480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.685480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:02.685480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:02.685480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.685480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:02.685480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:02.685480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.685480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:02.685480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:02.685480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.686080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:02.686080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:02.686080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:52:02.686080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:02.686080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:02.686080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.686080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:02.686080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:02.686080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:02.686080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.686080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:02.686080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:02.686080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:02.686080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:02.686080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:02.686080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:02.686080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:02.686080 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:02.686797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:02.686797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:02.686797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:02.686797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:02.686797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:02.686797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.686797 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:02.688525 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:02.688698 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:02.688803 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:02.688928 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:02.689326 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:02.689555 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.689992 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:02.689992 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:02.689992 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:02.689992 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:02.689992 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:02.689992 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:02.689992 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.689992 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:02.690560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.690560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:02.690560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:02.690560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:02.690560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.690560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:02.690560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:02.690560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:02.690560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.690560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:02.690560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:02.690560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:02.690560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:02.690560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:02.690560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:02.690876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:02.690876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:02.690876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:02.690876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:02.690876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:02.690876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:02.690876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:02.690876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.690876 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:02.691192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.691192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:02.691192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:02.691192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:02.691192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.691192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:02.691192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:02.691192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:02.691192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:02.691192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.691192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:02.691681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:02.691681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:02.691681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.691681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:02.691681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:02.691681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:02.691681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:02.691681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:02.691681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:02.691681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:02.691681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:02.691681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:02.691681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:02.691681 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:02.692193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:02.692193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:02.692193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.692193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:02.692193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.692193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:02.692193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:02.692193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:02.692193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:02.692193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:02.692193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.692193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:02.692193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:02.692193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.692473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:02.692473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:02.692473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.692473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:02.692473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:02.692473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.692473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:02.692473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:02.692473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:02.692473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:02.692473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.692473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:02.692473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:02.692473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:02.692473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.692473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:02.692473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:02.692473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:02.692473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:02.692757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:02.692757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:02.692757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:02.692757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:02.692757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:02.692757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:02.692757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:02.692757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:02.692757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:02.692757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.692757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:02.695498 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:02.695697 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:02.695778 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:02.695917 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:02.696382 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:02.696580 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:02.696694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.697171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:02.697171 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:02.697171 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:02.697171 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:02.697171 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:02.697171 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:02.697171 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.697171 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:02.698725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.698725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:02.698725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:02.698725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:02.698725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.698725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:02.698725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:02.698725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.698725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:02.698725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:02.698725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:02.698725 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.699042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:02.699042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:02.699042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:02.699042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:02.699042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:02.699042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:02.699042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:02.699042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:02.699042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:02.699042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:02.699042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:02.699042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:02.699042 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:02.699318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.699318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:02.699318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.699318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:02.699318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:02.699318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:02.699318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:02.699318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:02.699318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.699318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:02.699318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:02.699318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.699318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:02.699318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:02.699318 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.699569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:02.699569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:02.699569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:02.699569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:02.699569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:02.699569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.699569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:02.699569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:02.699569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:02.699569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:02.699569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:02.699569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:02.699569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:02.699569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:02.699569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:02.699569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:02.699569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:02.699569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:02.699569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:02.699885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:02.699885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:02.699885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:02.699885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:02.699885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:02.699885 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:03.880323 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:03.880948 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:03.880948 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:03.881600 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:03.914525 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:03.914746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:03.915322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:03.915322 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:03.915322 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:03.915322 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:03.915322 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:03.915322 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:03.915322 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:03.915322 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:03.916311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:03.916311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:03.916311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:03.916311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:03.916311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:03.916311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:03.916311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:03.916311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:03.916311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:03.916311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:03.916311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:03.916311 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:03.916559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:03.916559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:03.916559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:03.916559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:03.916559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:03.916559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:03.916559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:03.916559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:03.916559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:03.916559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:03.916559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:03.916559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:03.916559 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:03.916772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:03.916772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:03.917516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:03.917516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:03.917516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:03.917516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:03.917516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:03.917516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:03.917516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:03.917516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:03.917516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:03.917516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:03.917516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:03.917516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:03.917516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:03.917516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:03.917516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:03.917516 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:03.917947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:03.917947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:03.917947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:03.917947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:03.917947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:03.917947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:03.917947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:03.917947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:03.917947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:03.917947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:03.917947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:03.917947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:03.917947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:03.917947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:03.917947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:03.917947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:03.917947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:03.917947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:03.917947 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:03.918239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:03.918239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:03.918239 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:07.678153 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:07.678760 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:07.678918 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:07.679526 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:07.680346 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.680512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:07.680512 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:07.680512 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:07.680512 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:07.680512 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:07.680512 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:07.680512 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.680512 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:07.681214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.681214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:07.681214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:07.681214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:07.681214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.681214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:07.681214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:07.681214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.681214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:07.681214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:07.681214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:07.681214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.681528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:07.681528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:07.681528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:07.681528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:07.681528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:07.681528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:07.681528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:07.681528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:07.681528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:07.681528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:07.681528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:07.681528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:07.681528 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:07.681887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.681887 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:07.682216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.682216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:07.682216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:07.682216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:07.682216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:07.682216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:07.682216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.682216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:07.682216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:07.682216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.682216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:07.682216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:07.682216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.682216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:07.682216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:07.682216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.683109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:07.683109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:07.683109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.683109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:07.683109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:07.683109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:07.683109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.683109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:07.683109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:07.683109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:07.683109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:07.683109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:07.683109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:07.683109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:07.683109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:07.683109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:07.683109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:07.683109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:07.683109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:07.683864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:07.683864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.683864 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:07.683864 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:07.683864 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:07.683864 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:07.683864 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:52:07.683864 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:07.684098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.684098 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:07.684098 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:07.684098 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:07.684098 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:07.684098 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:07.684098 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:07.684098 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.684098 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:07.684825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.684825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:07.684825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:07.684825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:07.684825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.684825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:07.684825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:07.684825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:07.684825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.684825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:07.684825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:07.684825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:07.684825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:07.684825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:07.684825 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:07.685196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:07.685196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:07.685196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:07.685196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:07.685196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:07.685196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:07.685196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:07.685196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.685196 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:07.685741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.685741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:07.685741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:07.685741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:07.685741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.685741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:07.685741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:07.685741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:52:07.685741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:07.685741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:07.685741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.685741 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:07.686384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:07.686384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:07.686384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.686384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:07.686384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:07.686384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:07.686384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:07.686384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:07.686384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:07.686384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:07.686384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:07.686384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:07.686384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:07.686384 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:07.686848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:07.686848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:07.686848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.686848 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:07.686918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.686918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:07.686918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:07.686918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:07.686918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:07.686918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:07.686918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.686918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:07.686918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:07.686918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.686918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:07.686918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:07.686918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.686918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:07.686918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:07.686918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.687156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:07.687156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:07.687156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:52:07.687156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:07.687156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:07.687156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.687156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:07.687156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:07.687156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:07.687156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.687156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:07.687156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:07.687156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:07.687156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:07.687156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:07.687156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:07.687156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:07.687156 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:07.687563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:07.687563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:07.687563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:07.687563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:07.687563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:07.687563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.687563 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:07.688965 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:07.689387 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:07.689538 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:07.689715 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:07.690199 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:07.690520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.691083 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:07.691083 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:07.691083 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:07.691083 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:07.691083 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:07.691083 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:07.691083 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.691083 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:07.691821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.691821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:07.691821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:07.691821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:07.691821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.691821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:07.691821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:07.691821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:07.691821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.691821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:07.691821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:07.691821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:07.691821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:07.691821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:07.691821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:07.692094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:07.692094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:07.692094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:07.692094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:07.692094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:07.692094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:07.692094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:07.692094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.692094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:07.692568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.692568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:07.692568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:07.692568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:07.692568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.692568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:07.692568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:07.692568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:07.692568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:07.692568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.692568 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:07.693192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:07.693192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:07.693192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.693192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:07.693192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:07.693192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:07.693192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:07.693192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:07.693192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:07.693192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:07.693192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:07.693192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:07.693192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:07.693192 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:07.693806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:07.693806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:07.693806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.693806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:07.693806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.693806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:07.693806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:07.693806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:07.693806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:07.693806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:07.693806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.693806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:07.693806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:07.693806 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.694134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:07.694134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:07.694134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.694134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:07.694134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:07.694134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.694134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:07.694134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:07.694134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:07.694134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:07.694134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.694134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:07.694134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:07.694134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:07.694134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.694134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:07.694134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:07.694134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:07.694134 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:07.694653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:07.694653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:07.694653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:07.694653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:07.694653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:07.694653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:07.694653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:07.694653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:07.694653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:07.694653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.694653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:07.695942 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:07.696355 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:07.696481 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:07.696716 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:07.697093 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:07.697320 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:07.697426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.697965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:07.697965 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:07.697965 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:07.697965 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:07.697965 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:07.697965 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:07.697965 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.697965 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:07.698963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.698963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:07.698963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:07.698963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:07.698963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.698963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:07.698963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:07.698963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.698963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:07.698963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:07.698963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:07.698963 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.699403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:07.699403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:07.699403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:07.699403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:07.699403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:07.699403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:07.699403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:07.699403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:07.699403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:07.699403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:07.699403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:07.699403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:07.699403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:07.699846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.699846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:07.700076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.700076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:07.700076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:07.700076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:07.700076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:07.700076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:07.700076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.700076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:07.700076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:07.700076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.700076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:07.700076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:07.700076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.700076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:07.700076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:07.700076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:07.700629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:07.700629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:07.700629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.700629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:07.700629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:07.700629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:07.700629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:07.700629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:07.700629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:07.700629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:07.700629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:07.700629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:07.700629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:07.700629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:07.700629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:07.700629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:07.700629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:07.700629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:07.700629 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:07.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:07.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:07.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:08.880802 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:08.881279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:08.881279 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:08.881792 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:08.905958 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:08.906210 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:08.906783 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:08.906783 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:08.906783 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:08.906783 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:08.906783 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:08.906783 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:08.906783 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:08.906783 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:08.907487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:08.907487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:08.907487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:08.907487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:08.907487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:08.907487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:08.907487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:08.907487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:08.907487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:08.907487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:08.907487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:08.907487 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:08.907892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:08.907892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:08.907892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:08.907892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:08.907892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:08.907892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:08.907892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:08.907892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:08.907892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:08.907892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:08.907892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:08.907892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:08.907892 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:08.908275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:08.908275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:08.908434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:08.908434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:08.908434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:08.908434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:08.908434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:08.908434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:08.908434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:08.908434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:08.908434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:08.908434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:08.908434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:08.908434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:08.908434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:08.908434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:08.908434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:08.908434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:08.908884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:08.908884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:08.908884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:08.908884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:08.908884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:08.908884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:08.908884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:08.908884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:08.908884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:08.908884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:08.908884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:08.908884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:08.908884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:08.908884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:08.908884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:08.908884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:08.908884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:08.908884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:08.908884 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:08.909403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:08.909403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:08.909403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:12.678940 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:12.679604 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:12.679604 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:12.679793 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:12.679978 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.680529 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:12.680529 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:12.680529 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:12.680529 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:12.680529 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:12.680529 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:12.680529 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.680529 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:12.681369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.681369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:12.681369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:12.681369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:12.681369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.681369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:12.681369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:12.681369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.681369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:12.681369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:12.681369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:12.681369 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.681743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:12.681743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:12.681743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:12.681743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:12.681743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:12.681743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:12.681743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:12.681743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:12.681743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:12.681743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:12.681743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:12.681743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:12.681743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:12.682041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.682041 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:12.682319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.682319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:12.682319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:12.682319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:12.682319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:12.682319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:12.682319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.682319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:12.682319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:12.682319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.682319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:12.682319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:12.682319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.682319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:12.682319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:12.682319 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.683169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:12.683169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:12.683169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.683169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:12.683169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:12.683169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:12.683169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.683169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:12.683169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:12.683169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:12.683169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:12.683169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:12.683169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:12.683169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:12.683169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:12.683169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:12.683169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:12.683169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:12.683169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:12.683649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:12.683649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.683649 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:12.683649 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:12.683649 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:12.683649 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:12.683649 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:52:12.683649 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:12.683819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.684151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:12.684151 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:12.684151 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:12.684151 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:12.684151 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:12.684151 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:12.684151 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.684151 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:12.684777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.684777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:12.684777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:12.684777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:12.684777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.684777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:12.684777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:12.684777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:12.684777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.684777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:12.684777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:12.684777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:12.684777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:12.684777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:12.684777 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:12.685152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:12.685152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:12.685152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:12.685152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:12.685152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:12.685152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:12.685152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:12.685152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.685152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:12.685411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.685411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:12.685411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:12.685411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:12.685411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.685411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:12.685411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:12.685411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:52:12.685411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:12.685411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:12.685411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.685411 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:12.685840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:12.685840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:12.685840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.685840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:12.685840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:12.685840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:12.685840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:12.685840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:12.685840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:12.685840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:12.685840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:12.685840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:12.685840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:12.685840 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:12.686357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:12.686357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:12.686357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.686357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:12.686357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.686357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:12.686357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:12.686357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:12.686357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:12.686357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:12.686357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.686357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:12.686357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:12.686357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.686819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:12.686819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:12.686819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.686819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:12.686819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:12.686819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.686819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:12.686819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:12.686819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:52:12.686819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:12.686819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:12.686819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.686819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:12.686819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:12.686819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:12.686819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.686819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:12.686819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:12.686819 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:12.687204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:12.687204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:12.687204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:12.687204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:12.687204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:12.687204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:12.687204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:12.687204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:12.687204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:12.687204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:12.687204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.687204 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:12.689475 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:12.689673 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:12.689773 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:12.689864 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:12.690136 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:12.690386 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.690930 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:12.690930 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:12.690930 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:12.690930 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:12.690930 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:12.690930 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:12.690930 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.690930 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:12.691506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.691506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:12.691506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:12.691506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:12.691506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.691506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:12.691506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:12.691506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:12.691506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.691506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:12.691506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:12.691506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:12.691506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:12.691506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:12.691506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:12.691972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:12.691972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:12.691972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:12.691972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:12.691972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:12.691972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:12.691972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:12.691972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.691972 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:12.692199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.692199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:12.692199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:12.692199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:12.692199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.692199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:12.692199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:12.692199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:12.692199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:12.692199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.692199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:12.692469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:12.692469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:12.692469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.692469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:12.692469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:12.692469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:12.692469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:12.692469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:12.692469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:12.692469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:12.692469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:12.692469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:12.692469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:12.692469 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:12.692691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:12.692691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:12.692691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.692691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:12.692979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.692979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:12.692979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:12.692979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:12.692979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:12.692979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:12.692979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.692979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:12.692979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:12.692979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.692979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:12.692979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:12.692979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.692979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:12.692979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:12.692979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.692979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:12.693280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:12.693280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:12.693280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:12.693280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.693280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:12.693280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:12.693280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:12.693280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.693280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:12.693280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:12.693280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:12.693280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:12.693280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:12.693280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:12.693280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:12.693280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:12.693280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:12.693280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:12.693280 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:12.693717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:12.693717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:12.693717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.693717 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:12.696472 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:12.696732 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:12.696883 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:12.697058 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:12.697403 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:12.697636 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:12.697746 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.698233 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:12.698233 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:12.698233 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:12.698233 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:12.698233 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:12.698233 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:12.698233 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.698233 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:12.698902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.698902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:12.698902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:12.698902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:12.698902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.698902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:12.698902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:12.698902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.698902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:12.698902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:12.698902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:12.698902 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.699169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:12.699169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:12.699169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:12.699169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:12.699169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:12.699169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:12.699169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:12.699169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:12.699169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:12.699169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:12.699169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:12.699169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:12.699169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:12.699501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.699501 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:12.699878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.699878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:12.699878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:12.699878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:12.699878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:12.699878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:12.699878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.699878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:12.699878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:12.699878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.699878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:12.699878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:12.699878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.699878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:12.699878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:12.699878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:12.700400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:12.700400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:12.700400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.700400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:12.700400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:12.700400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:12.700400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:12.700400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:12.700400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:12.700400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:12.700400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:12.700400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:12.700400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:12.700400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:12.700400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:12.700400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:12.700400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:12.700400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:12.700400 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:12.700755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:12.700755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:12.700755 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:13.881153 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:13.881756 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:13.881756 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:13.882394 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:13.910130 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:13.910368 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:13.910865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:13.910865 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:13.910865 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:13.910865 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:13.910865 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:13.910865 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:13.910865 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:13.910865 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:13.911611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:13.911611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:13.911611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:13.911611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:13.911611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:13.911611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:13.911611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:13.911611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:13.911611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:13.911611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:13.911611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:13.911611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:13.911942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:13.911942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:13.911942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:13.911942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:13.911942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:13.911942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:13.911942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:13.911942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:13.911942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:13.911942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:13.911942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:13.911942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:13.911942 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:13.912206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:13.912206 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:13.912363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:13.912363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:13.912363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:13.912363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:13.912363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:13.912363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:13.912363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:13.912363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:13.912363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:13.912363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:13.912363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:13.912363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:13.912363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:13.912363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:13.912363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:13.912363 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:13.912772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:13.912772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:13.912772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:13.912772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:13.912772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:13.912772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:13.912772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:13.912772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:13.912772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:13.912772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:13.912772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:13.912772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:13.912772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:13.912772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:13.912772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:13.912772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:13.912772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:13.912772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:13.912772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:13.913268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:13.913268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:13.913268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:17.679278 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:17.679780 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:17.679780 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:17.680375 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:17.680583 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.681078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:17.681078 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:17.681078 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:17.681078 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:17.681078 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:17.681078 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:17.681078 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.681078 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:17.681795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.681795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:17.681795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:17.681795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:17.681795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.681795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:17.681795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:17.681795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.681795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:17.681795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:17.681795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:17.681795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.682152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:17.682152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:17.682152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:17.682152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:17.682152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:17.682152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:17.682152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:17.682152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:17.682152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:17.682152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:17.682152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:17.682152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:17.682152 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:17.682377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.682377 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:17.682513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.682513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:17.682513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:17.682513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:17.682513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:17.682513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:17.682513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.682513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:17.682513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:17.682513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.682513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:17.682513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:17.682513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.682513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:17.682513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:17.682513 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:17.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:17.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:17.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:17.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:17.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:17.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:17.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:17.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:17.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:17.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:17.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:17.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:17.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:17.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:17.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:17.683362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:17.683949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:17.683949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.683949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:17.683949 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:17.683949 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:17.683949 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:17.683949 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:52:17.683949 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:17.684171 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.684426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:17.684426 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:17.684426 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:17.684426 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:17.684426 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:17.684426 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:17.684426 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.684426 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:17.684877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.684877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:17.684877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:17.684877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:17.684877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.684877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:17.684877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:17.684877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:17.684877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.684877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:17.684877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:17.684877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:17.684877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:17.684877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:17.684877 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:17.685564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:17.685564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:17.685564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:17.685564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:17.685564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:17.685564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:17.685564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:17.685564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.685564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:17.685564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.685564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:17.685564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:17.685564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:17.685564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.686061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:17.686061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:17.686061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:52:17.686061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:17.686061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:17.686061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.686061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:17.686061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:17.686061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:17.686061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.686061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:17.686061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:17.686061 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:17.686302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:17.686302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:17.686302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:17.686302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:17.686302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:17.686302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:17.686302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:17.686302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:17.686302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:17.686302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:17.686302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.686302 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:17.686493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.686493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:17.686493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:17.686493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:17.686493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:17.686493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:17.686493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.686493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:17.686493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:17.686493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.686493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:17.686493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:17.686493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.686493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:17.686493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:17.686493 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.686778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:17.686778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:17.686778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:52:17.686778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:17.686778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:17.686778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.686778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:17.686778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:17.686778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:17.686778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.686778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:17.686778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:17.686778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:17.686778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:17.686778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:17.686778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:17.686778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:17.686778 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:17.687075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:17.687075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:17.687075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:17.687075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:17.687075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:17.687075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.687075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:17.689739 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:17.689947 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:17.690024 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:17.690125 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:17.690457 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:17.690705 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.691120 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:17.691120 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:17.691120 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:17.691120 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:17.691120 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:17.691120 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:17.691120 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.691120 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:17.691971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.691971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:17.691971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:17.691971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:17.691971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.691971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:17.691971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:17.691971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:17.691971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.691971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:17.691971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:17.691971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:17.691971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:17.691971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:17.691971 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:17.692279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:17.692279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:17.692279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:17.692279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:17.692279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:17.692279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:17.692279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:17.692279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.692279 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:17.692923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.692923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:17.692923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:17.692923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:17.692923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.692923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:17.692923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:17.692923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:17.692923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:17.692923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.692923 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:17.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:17.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:17.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:17.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:17.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:17.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:17.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:17.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:17.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:17.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:17.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:17.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:17.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:17.693370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:17.693370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:17.693370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.693370 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:17.694195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.694195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:17.694195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:17.694195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:17.694195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:17.694195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:17.694195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.694195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:17.694195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:17.694195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.694195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:17.694195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:17.694195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.694195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:17.694195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:17.694195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.694195 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:17.694542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:17.694542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:17.694542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:17.694542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.694542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:17.694542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:17.694542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:17.694542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.694542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:17.694542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:17.694542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:17.694542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:17.694542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:17.694542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:17.694542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:17.694542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:17.694542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:17.694542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:17.694542 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:17.694894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:17.694894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:17.694894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.694894 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:17.696702 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:17.696917 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:17.697035 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:17.697211 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:17.697495 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:17.697703 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:17.697812 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.698216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:17.698216 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:17.698216 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:17.698216 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:17.698216 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:17.698216 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:17.698216 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.698216 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:17.699151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.699151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:17.699151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:17.699151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:17.699151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.699151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:17.699151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:17.699151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.699151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:17.699151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:17.699151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:17.699151 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.699413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:17.699413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:17.699413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:17.699413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:17.699413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:17.699413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:17.699413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:17.699413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:17.699413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:17.699413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:17.699413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:17.699413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:17.699413 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:17.699636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.699636 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:17.700227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.700227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:17.700227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:17.700227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:17.700227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:17.700227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:17.700227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.700227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:17.700227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:17.700227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.700227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:17.700227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:17.700227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.700227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:17.700227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:17.700227 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:17.700547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:17.700547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:17.700547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.700547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:17.700547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:17.700547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:17.700547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:17.700547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:17.700547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:17.700547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:17.700547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:17.700547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:17.700547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:17.700547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:17.700547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:17.700547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:17.700547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:17.700547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:17.700547 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:17.700865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:17.700865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:17.700865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:18.881551 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:18.882145 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:18.882145 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:18.882661 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:18.911410 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:18.911654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:18.912169 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:18.912169 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:18.912169 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:18.912169 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:18.912169 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:18.912169 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:18.912169 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:18.912169 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:18.912953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:18.912953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:18.912953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:18.912953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:18.912953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:18.912953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:18.912953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:18.912953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:18.912953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:18.912953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:18.912953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:18.912953 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:18.913203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:18.913203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:18.913203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:18.913203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:18.913203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:18.913203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:18.913203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:18.913203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:18.913203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:18.913203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:18.913203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:18.913203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:18.913203 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:18.913463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:18.913463 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:18.913730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:18.913730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:18.913730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:18.913730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:18.913730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:18.913730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:18.913730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:18.913730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:18.913730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:18.913730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:18.913730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:18.913730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:18.913730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:18.913730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:18.913730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:18.913730 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:18.914005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:18.914005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:18.914005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:18.914005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:18.914005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:18.914005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:18.914005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:18.914005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:18.914005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:18.914005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:18.914005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:18.914005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:18.914005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:18.914005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:18.914005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:18.914005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:18.914005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:18.914005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:18.914005 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:18.914322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:18.914322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:18.914322 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:22.679743 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:22.680472 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:22.680472 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:22.680664 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:22.680767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.681155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:22.681155 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:22.681155 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:22.681155 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:22.681155 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:22.681155 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:22.681155 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.681155 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:22.681870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.681870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:22.681870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:22.681870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:22.681870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.681870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:22.681870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:22.681870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.681870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:22.681870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:22.681870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:22.681870 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.682353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:22.682353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:22.682353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:22.682353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:22.682353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:22.682353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:22.682353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:22.682353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:22.682353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:22.682353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:22.682353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:22.682353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:22.682353 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:22.683140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.683140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:22.683140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.683140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:22.683140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:22.683140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:22.683140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:22.683140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:22.683140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.683140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:22.683140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:22.683140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.683140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:22.683140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:22.683140 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.683670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:22.683670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:22.683670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.683670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:22.683670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:22.683670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.683670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:22.683670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:22.683670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:22.683670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.683670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:22.683670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:22.683670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:22.683670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:22.683670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:22.683670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:22.683670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:22.683670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:22.683670 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:22.683974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:22.683974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:22.683974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:22.683974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:22.683974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.683974 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:22.683974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:22.683974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:22.683974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:22.683974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:52:22.684143 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:22.684143 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.684548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:22.684548 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:22.684548 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:22.684548 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:22.684548 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:22.684548 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:22.684548 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.684548 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:22.685116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.685116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:22.685116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:22.685116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:22.685116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.685116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:22.685116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:22.685116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:22.685116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.685116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:22.685116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:22.685116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:22.685116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:22.685116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:22.685116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:22.685426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:22.685426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:22.685426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:22.685426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:22.685426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:22.685426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:22.685426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:22.685426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.685426 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:22.685772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.685772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:22.685772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:22.685772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:22.685772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.685772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:22.685772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:22.685772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:52:22.685772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:22.685772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:22.685772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.685772 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:22.686355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:22.686355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:22.686355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.686355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:22.686355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:22.686355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:22.686355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:22.686355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:22.686355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:22.686355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:22.686355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:22.686355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:22.686355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:22.686355 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:22.686716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:22.686716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:22.686716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.686716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:22.686716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.686716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:22.686716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:22.686716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:22.686716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:22.686716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:22.686716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.686716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:22.686716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:22.686716 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.686937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:22.686937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:22.686937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.686937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:22.686937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:22.686937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.686937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:22.686937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:22.686937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:52:22.686937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:22.686937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:22.686937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.686937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:22.686937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:22.686937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:22.686937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.686937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:22.686937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:22.686937 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:22.687220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:22.687220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:22.687220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:22.687220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:22.687220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:22.687220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:22.687220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:22.687220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:22.687220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:22.687220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:22.687220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.687220 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:22.690008 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:22.690187 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:22.690283 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:22.690381 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:22.690628 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:22.690813 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.691155 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:22.691155 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:22.691155 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:22.691155 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:22.691155 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:22.691155 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:22.691155 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.691155 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:22.691671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.691671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:22.691671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:22.691671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:22.691671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.691671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:22.691671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:22.691671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:22.691671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.691671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:22.691671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:22.691671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:22.691671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:22.691671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:22.691671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:22.691915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:22.691915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:22.691915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:22.691915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:22.691915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:22.691915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:22.691915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:22.691915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.691915 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:22.692264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.692264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:22.692264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:22.692264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:22.692264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.692264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:22.692264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:22.692264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:22.692264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:22.692264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.692264 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:22.692489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:22.692489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:22.692489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.692489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:22.692489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:22.692489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:22.692489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:22.692489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:22.692489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:22.692489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:22.692489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:22.692489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:22.692489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:22.692489 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:22.692721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:22.692721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:22.692721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.692721 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:22.693034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.693034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:22.693034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:22.693034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:22.693034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:22.693034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:22.693034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.693034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:22.693034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:22.693034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.693034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:22.693034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:22.693034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.693034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:22.693034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:22.693034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.693034 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:22.693850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:22.693850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:22.693850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:22.693850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.693850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:22.693850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:22.693850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:22.693850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.693850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:22.693850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:22.693850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:22.693850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:22.693850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:22.693850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:22.693850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:22.693850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:22.693850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:22.693850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:22.693850 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:22.694464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:22.694464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:22.694464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.694464 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:22.697077 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:22.697465 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:22.697595 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:22.697790 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:22.698113 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:22.698368 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:22.698471 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.698921 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:22.698921 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:22.698921 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:22.698921 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:22.698921 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:22.698921 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:22.698921 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.698921 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:22.699855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.699855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:22.699855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:22.699855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:22.699855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.699855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:22.699855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:22.699855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.699855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:22.699855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:22.699855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:22.699855 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.700110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:22.700110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:22.700110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:22.700110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:22.700110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:22.700110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:22.700110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:22.700110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:22.700110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:22.700110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:22.700110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:22.700110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:22.700110 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:22.700403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.700403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:22.700878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.700878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:22.700878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:22.700878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:22.700878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:22.700878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:22.700878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.700878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:22.700878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:22.700878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.700878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:22.700878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:22.700878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.700878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:22.700878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:22.700878 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:22.701684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:22.701684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:22.701684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.701684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:22.701684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:22.701684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:22.701684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:22.701684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:22.701684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:22.701684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:22.701684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:22.701684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:22.701684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:22.701684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:22.701684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:22.701684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:22.701684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:22.701684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:22.701684 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:22.702320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:22.702320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:22.702320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:23.882042 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:23.882582 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:23.882686 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:23.883507 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:23.914333 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:23.914333 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:23.914622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:23.914622 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:23.914622 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:23.914622 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:23.914622 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:23.914622 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:23.914622 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:23.914622 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:23.915350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:23.915350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:23.915350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:23.915350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:23.915350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:23.915350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:23.915350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:23.915350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:23.915350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:23.915350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:23.915350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:23.915350 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:23.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:23.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:23.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:23.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:23.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:23.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:23.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:23.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:23.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:23.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:23.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:23.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:23.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:23.916161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:23.916161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:23.916161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:23.916161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:23.916161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:23.916161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:23.916161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:23.916161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:23.916161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:23.916161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:23.916161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:23.916161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:23.916161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:23.916161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:23.916161 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:23.916668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:23.916668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:23.916668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:23.916668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:23.916668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:23.916668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:23.916668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:23.916668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:23.916668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:23.916668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:23.916668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:23.916668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:23.916668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:23.916668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:23.916668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:23.916668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:23.916668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:23.916668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:23.916668 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:23.917160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:23.917160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:23.917160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:23.917160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:23.917160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:23.917160 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:27.680238 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:27.680763 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:27.680763 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:27.681295 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:27.681548 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.682158 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:27.682158 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:27.682158 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:27.682158 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:27.682158 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:27.682158 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:27.682158 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.682158 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:27.682918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.682918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:27.682918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:27.682918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:27.682918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.682918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:27.682918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:27.682918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.682918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:27.682918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:27.682918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:27.682918 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.683270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:27.683270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:27.683270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:27.683270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:27.683270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:27.683270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:27.683270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:27.683270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:27.683270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:27.683270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:27.683270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:27.683270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:27.683270 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:27.683500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.683500 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:27.683739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.683739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:27.683739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:27.683739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:27.683739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:27.683739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:27.683739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.683739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:27.683739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:27.683739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.683739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:27.683739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:27.683739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.683739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:27.683739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:27.683739 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.684640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:27.684640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:27.684640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.684640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:27.684640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:27.684640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:27.684640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.684640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:27.684640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:27.684640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:27.684640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:27.684640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:27.684640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:27.684640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:27.684640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:27.684640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:27.684640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:27.684640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:27.684640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:27.685094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:27.685094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.685094 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:27.685094 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:27.685094 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:27.685094 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:27.685094 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:52:27.685094 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:27.685339 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.685466 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:27.685466 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:27.685466 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:27.685466 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:27.685466 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:27.685466 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:27.685466 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.685466 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:27.686084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.686084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:27.686084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:27.686084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:27.686084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.686084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:27.686084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:27.686084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:27.686084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.686084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:27.686084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:27.686084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:27.686084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:27.686084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:27.686084 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:27.686722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:27.686722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:27.686722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:27.686722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:27.686722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:27.686722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:27.686722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:27.686722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.686722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:27.686722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.686722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:27.686722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:27.686722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:27.686722 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.687326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:27.687326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:27.687326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:52:27.687326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:27.687326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:27.687326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.687326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:27.687326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:27.687326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:27.687326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.687326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:27.687326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:27.687326 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:27.687940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:27.687940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:27.687940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:27.687940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:27.687940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:27.687940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:27.687940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:27.687940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:27.687940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:27.687940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:27.687940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.687940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:27.688267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.688267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:27.688267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:27.688267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:27.688267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:27.688267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:27.688267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.688267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:27.688267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:27.688267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.688267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:27.688267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:27.688267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.688267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:27.688267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:27.688267 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.688550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:27.688550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:27.688550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:52:27.688550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:27.688550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:27.688550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.688550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:27.688550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:27.688550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:27.688550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.688550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:27.688550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:27.688550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:27.688550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:27.688550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:27.688550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:27.688550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:27.688550 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:27.688831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:27.688831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:27.688831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:27.688831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:27.688831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:27.688831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.688831 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:27.690458 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:27.690671 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:27.690769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:27.690880 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:27.691244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:27.691512 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.692079 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:27.692079 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:27.692079 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:27.692079 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:27.692079 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:27.692079 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:27.692079 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.692079 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:27.692913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.692913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:27.692913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:27.692913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:27.692913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.692913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:27.692913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:27.692913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:27.692913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.692913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:27.692913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:27.692913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:27.692913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:27.692913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:27.692913 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:27.693336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:27.693336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:27.693336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:27.693336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:27.693336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:27.693336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:27.693336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:27.693336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.693336 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:27.693569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.693569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:27.693569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:27.693569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:27.693569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.693569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:27.693569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:27.693569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:27.693569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:27.693569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.693569 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:27.693852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:27.693852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:27.693852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.693852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:27.693852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:27.693852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:27.693852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:27.693852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:27.693852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:27.693852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:27.693852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:27.693852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:27.693852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:27.693852 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:27.694105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:27.694105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:27.694105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.694105 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:27.694639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.694639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:27.694639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:27.694639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:27.694639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:27.694639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:27.694639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.694639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:27.694639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:27.694639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.694639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:27.694639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:27.694639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.694639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:27.694639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:27.694639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.694639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:27.695914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:27.695914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:27.695914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:27.695914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.695914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:27.695914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:27.695914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:27.695914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.695914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:27.695914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:27.695914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:27.695914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:27.695914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:27.695914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:27.695914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:27.695914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:27.695914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:27.695914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:27.695914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:27.696531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:27.696531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:27.696531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.696531 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:27.697448 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:27.697627 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:27.697717 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:27.697848 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:27.698228 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:27.698452 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:27.698578 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.698967 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:27.698967 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:27.698967 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:27.698967 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:27.698967 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:27.698967 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:27.698967 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.698967 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:27.699685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.699685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:27.699685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:27.699685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:27.699685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.699685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:27.699685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:27.699685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.699685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:27.699685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:27.699685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:27.699685 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.699880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:27.699880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:27.699880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:27.699880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:27.699880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:27.699880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:27.699880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:27.699880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:27.699880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:27.699880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:27.699880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:27.699880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:27.699880 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:27.700109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.700109 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:27.700367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.700367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:27.700367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:27.700367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:27.700367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:27.700367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:27.700367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.700367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:27.700367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:27.700367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.700367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:27.700367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:27.700367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.700367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:27.700367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:27.700367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:27.700653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:27.700653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:27.700653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.700653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:27.700653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:27.700653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:27.700653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:27.700653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:27.700653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:27.700653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:27.700653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:27.700653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:27.700653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:27.700653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:27.700653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:27.700653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:27.700653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:27.700653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:27.700653 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:27.700927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:27.700927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:27.700927 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:28.882681 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:28.883196 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:28.883432 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:28.884331 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:28.925169 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:28.925497 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:28.926222 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:28.926222 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:28.926222 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:28.926222 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:28.926222 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:28.926222 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:28.926222 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:28.926222 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:28.927056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:28.927056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:28.927056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:28.927056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:28.927056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:28.927056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:28.927056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:28.927056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:28.927056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:28.927056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:28.927056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:28.927056 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:28.927397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:28.927397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:28.927397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:28.927397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:28.927397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:28.927397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:28.927397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:28.927397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:28.927397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:28.927397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:28.927397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:28.927397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:28.927397 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:28.927631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:28.927631 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:28.927949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:28.927949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:28.927949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:28.927949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:28.927949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:28.927949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:28.927949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:28.927949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:28.927949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:28.927949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:28.927949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:28.927949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:28.927949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:28.927949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:28.927949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:28.927949 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:28.928420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:28.928420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:28.928420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:28.928420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:28.928420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:28.928420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:28.928420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:28.928420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:28.928420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:28.928420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:28.928420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:28.928420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:28.928420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:28.928420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:28.928420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:28.928420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:28.928420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:28.928420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:28.928420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:28.928973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:28.928973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:28.928973 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:32.680712 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:32.681170 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:32.681170 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:32.681675 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:32.681821 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:32.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:32.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:32.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:32.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:32.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:32.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.682223 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:32.682875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.682875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:32.682875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:32.682875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:32.682875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.682875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:32.682875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:32.682875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.682875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:32.682875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:32.682875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:32.682875 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.683286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:32.683286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:32.683286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:32.683286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:32.683286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:32.683286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:32.683286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:32.683286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:32.683286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:32.683286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:32.683286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:32.683286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:32.683286 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:32.683671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.683671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:32.683671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.683671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:32.683671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:32.683671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:32.683671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:32.683671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:32.683671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.683671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:32.683671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:32.683671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.683671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:32.683671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:32.683671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.684114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:32.684114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:32.684114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.684114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:32.684114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:32.684114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.684114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:32.684114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:32.684114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:32.684114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.684114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:32.684114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:32.684114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:32.684114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:32.684114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:32.684114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:32.684114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:32.684114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:32.684114 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:32.684620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:32.684620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:32.684620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:32.684620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:32.684620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.684620 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:32.684620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:32.684620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:32.684620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:32.684620 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:52:32.684895 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:32.685477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.685599 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:32.685599 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:32.685599 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:32.685599 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:32.685599 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:32.685599 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:32.685599 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.685599 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:32.686091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.686091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:32.686091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:32.686091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:32.686091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.686091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:32.686091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:32.686091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:32.686091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.686091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:32.686091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:32.686091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:32.686091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:32.686091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:32.686091 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:32.686543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:32.686543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:32.686543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:32.686543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:32.686543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:32.686543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:32.686543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:32.686543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.686543 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:32.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:32.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:32.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:32.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:32.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:32.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:52:32.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:32.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:32.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:32.687881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:32.687881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:32.687881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.687881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:32.687881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:32.687881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:32.687881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:32.687881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:32.687881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:32.687881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:32.687881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:32.687881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:32.687881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:32.687881 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:32.688767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:32.688767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:32.688767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.688767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:32.688767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.688767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:32.688767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:32.688767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:32.688767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:32.688767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:32.688767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.688767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:32.688767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:32.688767 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.689627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:32.689627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:32.689627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.689627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:32.689627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:32.689627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.689627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:32.689627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:32.689627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:52:32.689627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:32.689627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:32.689627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.689627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:32.689627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:32.689627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:32.689627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.689627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:32.689627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:32.689627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:32.690667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:32.690667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:32.690667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:32.690667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:32.690667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:32.690667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:32.690667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:32.690667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:32.690667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:32.690667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:32.690667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.690667 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:32.691038 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:32.691038 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:32.691038 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:32.691038 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:32.691311 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:32.691479 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.691805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:32.691805 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:32.691805 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:32.691805 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:32.691805 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:32.691805 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:32.691805 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.691805 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:32.692269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.692269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:32.692269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:32.692269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:32.692269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.692269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:32.692269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:32.692269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:32.692269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.692269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:32.692269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:32.692269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:32.692269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:32.692269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:32.692269 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:32.692759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:32.692759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:32.692759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:32.692759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:32.692759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:32.692759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:32.692759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:32.692759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.692759 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:32.693008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.693008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:32.693008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:32.693008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:32.693008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.693008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:32.693008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:32.693008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:32.693008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:32.693008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.693008 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:32.693367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:32.693367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:32.693367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.693367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:32.693367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:32.693367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:32.693367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:32.693367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:32.693367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:32.693367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:32.693367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:32.693367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:32.693367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:32.693367 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:32.693756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:32.693756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:32.693756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.693756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:32.693756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.693756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:32.693756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:32.693756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:32.693756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:32.693756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:32.693756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.693756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:32.693756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:32.693756 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.694172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:32.694172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:32.694172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.694172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:32.694172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:32.694172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.694172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:32.694172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:32.694172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:32.694172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:32.694172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.694172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:32.694172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:32.694172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:32.694172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.694172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:32.694172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:32.694172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:32.694172 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:32.694656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:32.694656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:32.694656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:32.694656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:32.694656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:32.694656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:32.694656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:32.694656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:32.694656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:32.694656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.694656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:32.697695 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:32.697857 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:32.697921 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:32.698051 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:32.698263 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:32.698430 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:32.698507 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.698814 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:32.698814 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:32.698814 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:32.698814 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:32.698814 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:32.698814 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:32.698814 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.698814 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:32.699361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.699361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:32.699361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:32.699361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:32.699361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.699361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:32.699361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:32.699361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.699361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:32.699361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:32.699361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:32.699361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.699713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:32.699713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:32.699713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:32.699713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:32.699713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:32.699713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:32.699713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:32.699713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:32.699713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:32.699713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:32.699713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:32.699713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:32.699713 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:32.699925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.699925 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:32.699994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.699994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:32.699994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:32.699994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:32.699994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:32.699994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:32.699994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.699994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:32.699994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:32.699994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.699994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:32.699994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:32.699994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.699994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:32.699994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:32.699994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:32.700294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:32.700294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:32.700294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.700294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:32.700294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:32.700294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:32.700294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:32.700294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:32.700294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:32.700294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:32.700294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:32.700294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:32.700294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:32.700294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:32.700294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:32.700294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:32.700294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:32.700294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:32.700294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:32.700844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:32.700844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:32.700844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:33.883185 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:33.883807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:33.883807 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:33.884509 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:33.909853 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:33.910064 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:33.910591 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:33.910591 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:33.910591 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:33.910591 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:33.910591 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:33.910591 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:33.910591 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:33.910591 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:33.911238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:33.911238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:33.911238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:33.911238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:33.911238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:33.911238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:33.911238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:33.911238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:33.911238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:33.911238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:33.911238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:33.911238 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:33.911669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:33.911669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:33.911669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:33.911669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:33.911669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:33.911669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:33.911669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:33.911669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:33.911669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:33.911669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:33.911669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:33.911669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:33.911669 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:33.912026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:33.912026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:33.912026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:33.912026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:33.912026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:33.912026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:33.912026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:33.912026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:33.912026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:33.912026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:33.912026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:33.912026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:33.912026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:33.912026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:33.912026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:33.912603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:33.912603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:33.912603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:33.912603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:33.912603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:33.912603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:33.912603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:33.912603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:33.912603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:33.912603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:33.912603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:33.912603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:33.912603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:33.912603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:33.912603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:33.912603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:33.912603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:33.912603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:33.912603 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:33.913177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:33.913177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:33.913177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:33.913177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:33.913177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:33.913177 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:37.681244 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:37.681726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:37.681726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:37.682185 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:37.682452 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.682996 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:37.682996 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:37.682996 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:37.682996 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:37.682996 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:37.682996 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:37.682996 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.682996 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:37.683788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.683788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:37.683788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:37.683788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:37.683788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.683788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:37.683788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:37.683788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.683788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:37.683788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:37.683788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:37.683788 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.683998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:37.683998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:37.683998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:37.683998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:37.683998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:37.683998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:37.683998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:37.683998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:37.683998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:37.683998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:37.683998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:37.683998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:37.683998 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:37.684236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.684236 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:37.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:37.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:37.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:37.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:37.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:37.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:37.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:37.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:37.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:37.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:37.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:37.684680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.684990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:37.684990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:37.684990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.684990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:37.684990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:37.684990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:37.684990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.684990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:37.684990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:37.684990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:37.684990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:37.684990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:37.684990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:37.684990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:37.684990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:37.684990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:37.684990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:37.684990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:37.684990 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:37.685362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:37.685362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.685362 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:37.685362 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:37.685362 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:37.685362 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:37.685362 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:52:37.685545 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:37.685743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.686207 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:37.686207 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:37.686207 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:37.686207 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:37.686207 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:37.686207 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:37.686207 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.686207 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:37.686824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.686824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:37.686824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:37.686824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:37.686824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.686824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:37.686824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:37.686824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:37.686824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.686824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:37.686824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:37.686824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:37.686824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:37.686824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:37.686824 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:37.687088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:37.687088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:37.687088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:37.687088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:37.687088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:37.687088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:37.687088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:37.687088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.687088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:37.687533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.687533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:37.687533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:37.687533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:37.687533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.687533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:37.687533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:37.687533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:52:37.687533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:37.687533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:37.687533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.687533 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:37.687765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:37.687765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:37.687765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.687765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:37.687765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:37.687765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:37.687765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:37.687765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:37.687765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:37.687765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:37.687765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:37.687765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:37.687765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:37.687765 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:37.688015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:37.688015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:37.688015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.688015 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:37.688306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.688306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:37.688306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:37.688306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:37.688306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:37.688306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:37.688306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.688306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:37.688306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:37.688306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.688306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:37.688306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:37.688306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.688306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:37.688306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:37.688306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.688622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:37.688622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:37.688622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:52:37.688622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:37.688622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:37.688622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.688622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:37.688622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:37.688622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:37.688622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.688622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:37.688622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:37.688622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:37.688622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:37.688622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:37.688622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:37.688622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:37.688622 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:37.688993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:37.688993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:37.688993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:37.688993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:37.688993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:37.688993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.688993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:37.690974 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:37.691145 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:37.691236 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:37.691357 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:37.691711 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:37.691911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.692379 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:37.692379 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:37.692379 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:37.692379 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:37.692379 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:37.692379 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:37.692379 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.692379 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:37.693069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.693069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:37.693069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:37.693069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:37.693069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.693069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:37.693069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:37.693069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:37.693069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.693069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:37.693069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:37.693069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:37.693069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:37.693069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:37.693069 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:37.693473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:37.693473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:37.693473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:37.693473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:37.693473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:37.693473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:37.693473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:37.693473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.693473 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:37.693982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.693982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:37.693982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:37.693982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:37.693982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.693982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:37.693982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:37.693982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:37.693982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:37.693982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.693982 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:37.694314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:37.694314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:37.694314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.694314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:37.694314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:37.694314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:37.694314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:37.694314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:37.694314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:37.694314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:37.694314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:37.694314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:37.694314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:37.694314 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:37.694691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:37.694691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:37.694691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.694691 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:37.695214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.695214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:37.695214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:37.695214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:37.695214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:37.695214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:37.695214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.695214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:37.695214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:37.695214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.695214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:37.695214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:37.695214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.695214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:37.695214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:37.695214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.695214 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:37.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:37.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:37.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:37.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:37.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:37.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:37.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:37.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:37.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:37.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:37.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:37.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:37.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:37.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:37.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:37.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:37.695510 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:37.695833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:37.695833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:37.695833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.695833 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:37.697952 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:37.698185 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:37.698326 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:37.698495 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:37.698769 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:37.698965 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:37.699057 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.699455 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:37.699455 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:37.699455 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:37.699455 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:37.699455 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:37.699455 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:37.699455 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.699455 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:37.700193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.700193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:37.700193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:37.700193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:37.700193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.700193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:37.700193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:37.700193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.700193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:37.700193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:37.700193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:37.700193 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.700540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:37.700540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:37.700540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:37.700540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:37.700540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:37.700540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:37.700540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:37.700540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:37.700540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:37.700540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:37.700540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:37.700540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:37.700540 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:37.700941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.700941 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:37.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:37.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:37.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:37.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:37.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:37.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:37.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:37.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:37.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:37.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:37.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:37.701200 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:37.701502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:37.701502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:37.701502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.701502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:37.701502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:37.701502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:37.701502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:37.701502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:37.701502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:37.701502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:37.701502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:37.701502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:37.701502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:37.701502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:37.701502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:37.701502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:37.701502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:37.701502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:37.701502 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:37.701837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:37.701837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:37.701837 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:38.883619 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:38.884074 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:38.884176 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:38.884925 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:38.912157 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:38.912412 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:38.912975 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:38.912975 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:38.912975 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:38.912975 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:38.912975 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:38.912975 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:38.912975 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:38.912975 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:38.913859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:38.913859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:38.913859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:38.913859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:38.913859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:38.913859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:38.913859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:38.913859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:38.913859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:38.913859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:38.913859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:38.913859 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:38.914234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:38.914234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:38.914234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:38.914234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:38.914234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:38.914234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:38.914234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:38.914234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:38.914234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:38.914234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:38.914234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:38.914234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:38.914234 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:38.914637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:38.914637 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:38.914712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:38.914712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:38.914712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:38.914712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:38.914712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:38.914712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:38.914712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:38.914712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:38.914712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:38.914712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:38.914712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:38.914712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:38.914712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:38.914712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:38.914712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:38.914712 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:38.915145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:38.915145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:38.915145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:38.915145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:38.915145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:38.915145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:38.915145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:38.915145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:38.915145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:38.915145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:38.915145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:38.915145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:38.915145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:38.915145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:38.915145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:38.915145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:38.915145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:38.915145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:38.915145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:38.915707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:38.915707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:38.915707 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:42.681656 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:42.682157 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:42.682157 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:42.682469 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:42.682628 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.683048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:42.683048 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:42.683048 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:42.683048 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:42.683048 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:42.683048 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:42.683048 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.683048 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:42.683687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.683687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:42.683687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:42.683687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:42.683687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.683687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:42.683687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:42.683687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.683687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:42.683687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:42.683687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:42.683687 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.684003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:42.684003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:42.684003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:42.684003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:42.684003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:42.684003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:42.684003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:42.684003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:42.684003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:42.684003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:42.684003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:42.684003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:42.684003 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:42.684202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.684202 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:42.684552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.684552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:42.684552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:42.684552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:42.684552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:42.684552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:42.684552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.684552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:42.684552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:42.684552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.684552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:42.684552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:42.684552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.684552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:42.684552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:42.684552 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.685174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:42.685174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:42.685174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.685174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:42.685174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:42.685174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:42.685174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.685174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:42.685174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:42.685174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:42.685174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:42.685174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:42.685174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:42.685174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:42.685174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:42.685174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:42.685174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:42.685174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:42.685174 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:42.685523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:42.685523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.685523 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:42.685523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:42.685523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:42.685523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:42.685523 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:52:42.685668 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:42.685993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.686067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:42.686067 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:42.686067 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:42.686067 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:42.686067 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:42.686067 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:42.686067 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.686067 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:42.686509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.686509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:42.686509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:42.686509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:42.686509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.686509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:42.686509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:42.686509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:42.686509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.686509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:42.686509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:42.686509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:42.686509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:42.686509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:42.686509 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:42.687085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:42.687085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:42.687085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:42.687085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:42.687085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:42.687085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:42.687085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:42.687085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.687085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:42.687085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.687085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:42.687085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:42.687085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:42.687085 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:42.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:42.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:52:42.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:42.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:42.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:42.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:42.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:42.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:42.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:42.687416 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:42.687769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:42.687769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:42.687769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:42.687769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:42.687769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:42.687769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:42.687769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:42.687769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:42.687769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:42.687769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:42.687769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.687769 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:42.688067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.688067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:42.688067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:42.688067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:42.688067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:42.688067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:42.688067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.688067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:42.688067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:42.688067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.688067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:42.688067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:42.688067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.688067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:42.688067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:42.688067 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.688560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:42.688560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:42.688560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:52:42.688560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:42.688560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:42.688560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.688560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:42.688560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:42.688560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:42.688560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.688560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:42.688560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:42.688560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:42.688560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:42.688560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:42.688560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:42.688560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:42.688560 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:42.689107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:42.689107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:42.689107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:42.689107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:42.689107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:42.689107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.689107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:42.691203 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:42.691418 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:42.691584 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:42.691584 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:42.691815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:42.691965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.692294 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:42.692294 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:42.692294 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:42.692294 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:42.692294 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:42.692294 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:42.692294 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.692294 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:42.692751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.692751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:42.692751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:42.692751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:42.692751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.692751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:42.692751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:42.692751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:42.692751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.692751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:42.692751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:42.692751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:42.692751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:42.692751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:42.692751 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:42.692994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:42.692994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:42.692994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:42.692994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:42.692994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:42.692994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:42.692994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:42.692994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.692994 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:42.693395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.693395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:42.693395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:42.693395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:42.693395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.693395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:42.693395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:42.693395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:42.693395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:42.693395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.693395 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:42.693964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:42.693964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:42.693964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.693964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:42.693964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:42.693964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:42.693964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:42.693964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:42.693964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:42.693964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:42.693964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:42.693964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:42.693964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:42.693964 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:42.694162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:42.694162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:42.694162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.694162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:42.694162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.694162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:42.694162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:42.694162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:42.694162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:42.694162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:42.694162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.694162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:42.694162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:42.694162 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.694409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:42.694409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:42.694409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.694409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:42.694409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:42.694409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.694409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:42.694409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:42.694409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:42.694409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:42.694409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.694409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:42.694409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:42.694409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:42.694409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.694409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:42.694409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:42.694409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:42.694409 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:42.694735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:42.694735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:42.694735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:42.694735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:42.694735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:42.694735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:42.694735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:42.694735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:42.694735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:42.694735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.694735 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:42.698225 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:42.698511 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:42.698613 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:42.698726 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:42.698954 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:42.699105 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:42.699198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.699562 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:42.699562 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:42.699562 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:42.699562 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:42.699562 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:42.699562 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:42.699562 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.699562 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:42.700090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.700090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:42.700090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:42.700090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:42.700090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.700090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:42.700090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:42.700090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.700090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:42.700090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:42.700090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:42.700090 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.700320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:42.700320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:42.700320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:42.700320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:42.700320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:42.700320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:42.700320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:42.700320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:42.700320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:42.700320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:42.700320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:42.700320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:42.700320 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:42.700518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.700518 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:42.700811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.700811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:42.700811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:42.700811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:42.700811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:42.700811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:42.700811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.700811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:42.700811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:42.700811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.700811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:42.700811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:42.700811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.700811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:42.700811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:42.700811 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:42.701556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:42.701556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:42.701556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.701556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:42.701556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:42.701556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:42.701556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:42.701556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:42.701556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:42.701556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:42.701556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:42.701556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:42.701556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:42.701556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:42.701556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:42.701556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:42.701556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:42.701556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:42.701556 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:42.701879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:42.701879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:42.701879 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:43.884064 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:43.884644 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:43.884644 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:43.885163 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:43.911949 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:43.912191 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:43.912600 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:43.912600 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:43.912600 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:43.912600 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:43.912600 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:43.912600 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:43.912600 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:43.912600 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:43.913268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:43.913268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:43.913268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:43.913268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:43.913268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:43.913268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:43.913268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:43.913268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:43.913268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:43.913268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:43.913268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:43.913268 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:43.913514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:43.913514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:43.913514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:43.913514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:43.913514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:43.913514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:43.913514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:43.913514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:43.913514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:43.913514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:43.913514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:43.913514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:43.913514 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:43.913784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:43.913784 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:43.914106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:43.914106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:43.914106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:43.914106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:43.914106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:43.914106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:43.914106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:43.914106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:43.914106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:43.914106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:43.914106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:43.914106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:43.914106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:43.914106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:43.914106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:43.914106 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:43.914956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:43.914956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:43.914956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:43.914956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:43.914956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:43.914956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:43.914956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:43.914956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:43.914956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:43.914956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:43.914956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:43.914956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:43.914956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:43.914956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:43.914956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:43.914956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:43.914956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:43.914956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:43.914956 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:43.915437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:43.915437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:43.915437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:47.682134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:47.682621 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:47.682621 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:47.683096 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:47.683347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.683939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:47.683939 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:47.683939 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:47.683939 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:47.683939 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:47.683939 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:47.683939 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.683939 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:47.684917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.684917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:47.684917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:47.684917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:47.684917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.684917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:47.684917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:47.684917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.684917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:47.684917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:47.684917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:47.684917 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.685130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:47.685130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:47.685130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:47.685130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:47.685130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:47.685130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:47.685130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:47.685130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:47.685130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:47.685130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:47.685130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:47.685130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:47.685130 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:47.685347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.685347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:47.686037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.686037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:47.686037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:47.686037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:47.686037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:47.686037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:47.686037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.686037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:47.686037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:47.686037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.686037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:47.686037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:47.686037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.686037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:47.686037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:47.686037 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.686627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:47.686627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:47.686627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.686627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:47.686627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:47.686627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:47.686627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.686627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:47.686627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:47.686627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:47.686627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:47.686627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:47.686627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:47.686627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:47.686627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:47.686627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:47.686627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:47.686627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:47.686627 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:47.687052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:47.687052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.687052 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:47.687052 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:47.687052 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:47.687052 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:47.687052 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:52:47.687052 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:47.687199 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.687689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:47.687689 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:47.687689 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:47.687689 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:47.687689 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:47.687689 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:47.687689 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.687689 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:47.688437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.688437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:47.688437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:47.688437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:47.688437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.688437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:47.688437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:47.688437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:47.688437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.688437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:47.688437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:47.688437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:47.688437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:47.688437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:47.688437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:47.688676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:47.688676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:47.688676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:47.688676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:47.688676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:47.688676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:47.688676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:47.688676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.688676 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:47.689237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.689237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:47.689237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:47.689237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:47.689237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.689237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:47.689237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:47.689237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:52:47.689237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:47.689237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:47.689237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.689237 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:47.689451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:47.689451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:47.689451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.689451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:47.689451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:47.689451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:47.689451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:47.689451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:47.689451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:47.689451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:47.689451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:47.689451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:47.689451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:47.689451 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:47.689700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:47.689700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:47.689700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.689700 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:47.690276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.690276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:47.690276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:47.690276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:47.690276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:47.690276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:47.690276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.690276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:47.690276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:47.690276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.690276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:47.690276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:47.690276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.690276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:47.690276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:47.690276 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.690675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:47.690675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:47.690675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:52:47.690675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:47.690675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:47.690675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.690675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:47.690675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:47.690675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:47.690675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.690675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:47.690675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:47.690675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:47.690675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:47.690675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:47.690675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:47.690675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:47.690675 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:47.691065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:47.691065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:47.691065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:47.691065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:47.691065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:47.691065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.691065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:47.691386 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:47.691618 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:47.691753 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:47.691812 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:47.692219 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:47.692437 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.692910 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:47.692910 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:47.692910 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:47.692910 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:47.692910 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:47.692910 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:47.692910 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.692910 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:47.693656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.693656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:47.693656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:47.693656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:47.693656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.693656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:47.693656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:47.693656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:47.693656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.693656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:47.693656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:47.693656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:47.693656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:47.693656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:47.693656 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:47.693950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:47.693950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:47.693950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:47.693950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:47.693950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:47.693950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:47.693950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:47.693950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.693950 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:47.694506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.694506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:47.694506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:47.694506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:47.694506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.694506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:47.694506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:47.694506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:47.694506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:47.694506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.694506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:47.694702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:47.694702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:47.694702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.694702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:47.694702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:47.694702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:47.694702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:47.694702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:47.694702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:47.694702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:47.694702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:47.694702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:47.694702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:47.694702 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:47.694903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:47.694903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:47.694903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.694903 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:47.695564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.695564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:47.695564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:47.695564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:47.695564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:47.695564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:47.695564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.695564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:47.695564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:47.695564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.695564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:47.695564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:47.695564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.695564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:47.695564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:47.695564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.695564 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:47.695818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:47.695818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:47.695818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:47.695818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.695818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:47.695818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:47.695818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:47.695818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.695818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:47.695818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:47.695818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:47.695818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:47.695818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:47.695818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:47.695818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:47.695818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:47.695818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:47.695818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:47.695818 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:47.696089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:47.696089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:47.696089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.696089 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:47.698530 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:47.698707 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:47.698815 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:47.698966 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:47.699264 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:47.699493 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:47.699623 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.700139 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:47.700139 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:47.700139 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:47.700139 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:47.700139 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:47.700139 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:47.700139 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.700139 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:47.700977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.700977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:47.700977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:47.700977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:47.700977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.700977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:47.700977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:47.700977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.700977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:47.700977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:47.700977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:47.700977 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.701460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:47.701460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:47.701460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:47.701460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:47.701460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:47.701460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:47.701460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:47.701460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:47.701460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:47.701460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:47.701460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:47.701460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:47.701460 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:47.701775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.701775 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:47.702076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.702076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:47.702076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:47.702076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:47.702076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:47.702076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:47.702076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.702076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:47.702076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:47.702076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.702076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:47.702076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:47.702076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.702076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:47.702076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:47.702076 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:47.702658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:47.702658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:47.702658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.702658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:47.702658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:47.702658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:47.702658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:47.702658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:47.702658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:47.702658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:47.702658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:47.702658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:47.702658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:47.702658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:47.702658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:47.702658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:47.702658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:47.702658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:47.702658 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:47.703009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:47.703009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:47.703009 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:48.884607 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:48.885231 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:48.885231 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:48.885757 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:48.913198 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:48.913420 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:48.913889 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:48.913889 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:48.913889 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:48.913889 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:48.913889 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:48.913889 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:48.913889 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:48.913889 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:48.914748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:48.914748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:48.914748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:48.914748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:48.914748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:48.914748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:48.914748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:48.914748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:48.914748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:48.914748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:48.914748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:48.914748 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:48.915088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:48.915088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:48.915088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:48.915088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:48.915088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:48.915088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:48.915088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:48.915088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:48.915088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:48.915088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:48.915088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:48.915088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:48.915088 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:48.915434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:48.915434 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:48.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:48.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:48.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:48.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:48.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:48.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:48.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:48.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:48.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:48.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:48.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:48.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:48.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:48.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:48.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:48.915792 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:48.916108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:48.916108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:48.916108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:48.916108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:48.916108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:48.916108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:48.916108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:48.916108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:48.916108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:48.916108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:48.916108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:48.916108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:48.916108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:48.916108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:48.916108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:48.916108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:48.916108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:48.916108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:48.916108 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:48.916447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:48.916447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:48.916447 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:52.682755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:52.683359 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:52.683359 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:52.683768 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:52.683979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.684515 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:52.684515 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:52.684515 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:52.684515 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:52.684515 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:52.684515 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:52.684515 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.684515 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:52.685522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.685522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:52.685522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:52.685522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:52.685522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.685522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:52.685522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:52.685522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.685522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:52.685522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:52.685522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:52.685522 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.685957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:52.685957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:52.685957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:52.685957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:52.685957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:52.685957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:52.685957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:52.685957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:52.685957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:52.685957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:52.685957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:52.685957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:52.685957 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:52.686388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.686388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:52.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:52.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:52.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:52.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:52.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:52.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:52.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:52.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:52.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:52.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:52.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:52.687026 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.687962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:52.687962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:52.687962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.687962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:52.687962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:52.687962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:52.687962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.687962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:52.687962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:52.687962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:52.687962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:52.687962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:52.687962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:52.687962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:52.687962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:52.687962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:52.687962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:52.687962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:52.687962 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:52.688611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:52.688611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.688611 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:52.688611 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:52.688611 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:52.688611 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:52.688611 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:52:52.688611 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:52.688865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.688865 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:52.688865 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:52.688865 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:52.688865 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:52.688865 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:52.688865 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:52.688865 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.688865 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:52.689652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.689652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:52.689652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:52.689652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:52.689652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.689652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:52.689652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:52.689652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:52.689652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.689652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:52.689652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:52.689652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:52.689652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:52.689652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:52.689652 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:52.690065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:52.690065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:52.690065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:52.690065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:52.690065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:52.690065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:52.690065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:52.690065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.690065 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:52.690654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.690654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:52.690654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:52.690654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:52.690654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.690654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:52.690654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:52.690654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:52:52.690654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:52.690654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:52.690654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.690654 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:52.691361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:52.691361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:52.691361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.691361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:52.691361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:52.691361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:52.691361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:52.691361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:52.691361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:52.691361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:52.691361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:52.691361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:52.691361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:52.691361 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:52.692338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:52.692338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:52.692338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.692338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:52.692338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.692338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:52.692338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:52.692338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:52.692338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:52.692338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:52.692338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.692338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:52.692338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:52.692338 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.692699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:52.692699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:52.692699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.692699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:52.692699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:52.692699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.692699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:52.692699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:52.692699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:52:52.692699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:52.692699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:52.692699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.692699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:52.692699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:52.692699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:52.692699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.692699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:52.692699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:52.692699 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:52.693029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:52.693029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:52.693029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:52.693029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:52.693029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:52.693029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:52.693029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:52.693029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:52.693029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:52.693029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:52.693029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.693029 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:52.693029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:52.693029 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:52.693301 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:52.693301 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:52.693301 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:52.693301 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.693869 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:52.693869 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:52.693869 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:52.693869 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:52.693869 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:52.693869 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:52.693869 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.693869 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:52.694610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.694610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:52.694610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:52.694610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:52.694610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.694610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:52.694610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:52.694610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:52.694610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.694610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:52.694610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:52.694610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:52.694610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:52.694610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:52.694610 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:52.695582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:52.695582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:52.695582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:52.695582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:52.695582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:52.695582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:52.695582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:52.695582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.695582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:52.695582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.695582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:52.695582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:52.695582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:52.695582 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.696306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:52.696306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:52.696306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:52.696306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:52.696306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.696306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:52.696306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:52.696306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:52.696306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.696306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:52.696306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:52.696306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:52.696306 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:52.696745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:52.696745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:52.696745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:52.696745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:52.696745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:52.696745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:52.696745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:52.696745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:52.696745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:52.696745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.696745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:52.696745 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.696940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:52.696940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:52.696940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:52.696940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:52.696940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:52.696940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.696940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:52.696940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:52.696940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.696940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:52.696940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:52.696940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.696940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:52.696940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:52.696940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.696940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:52.696940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:52.696940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:52.696940 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.697296 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:52.698910 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:52.699124 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:52.699216 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:52.699390 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:52.699782 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:52.700025 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:52.700145 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.700732 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:52.700732 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:52.700732 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:52.700732 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:52.700732 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:52.700732 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:52.700732 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.700732 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:52.701520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.701520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:52.701520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:52.701520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:52.701520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.701520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:52.701520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:52.701520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.701520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:52.701520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:52.701520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:52.701520 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.701770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:52.701770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:52.701770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:52.701770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:52.701770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:52.701770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:52.701770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:52.701770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:52.701770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:52.701770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:52.701770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:52.701770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:52.701770 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:52.701993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.701993 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:52.702293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.702293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:52.702293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:52.702293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:52.702293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:52.702293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:52.702293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.702293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:52.702293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:52.702293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.702293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:52.702293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:52.702293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.702293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:52.702293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:52.702293 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:52.702743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:52.702743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:52.702743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.702743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:52.702743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:52.702743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:52.702743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:52.702743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:52.702743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:52.702743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:52.702743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:52.702743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:52.702743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:52.702743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:52.702743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:52.702743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:52.702743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:52.702743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:52.702743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:52.703078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:52.703078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:52.703078 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:53.885187 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:53.885710 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:53.885710 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:53.886384 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:53.896134 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG dbcounter [-] [89274] Writing DB stats neutron:SELECT=242 {{(pid=89274) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 14:52:53.913241 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:53.913586 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:53.914275 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:53.914275 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:53.914275 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:53.914275 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:53.914275 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:53.914275 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:53.914275 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:53.914275 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:53.915075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:53.915075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:53.915075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:53.915075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:53.915075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:53.915075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:53.915075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:53.915075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:53.915075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:53.915075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:53.915075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:53.915075 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:53.915390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:53.915390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:53.915390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:53.915390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:53.915390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:53.915390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:53.915390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:53.915390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:53.915390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:53.915390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:53.915390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:53.915390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:53.915390 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:53.915640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:53.915640 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:53.915795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:53.915795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:53.915795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:53.915795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:53.915795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:53.915795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:53.915795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:53.915795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:53.915795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:53.915795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:53.915795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:53.915795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:53.915795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:53.915795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:53.915795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:53.915795 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:53.916081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:53.916081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:53.916081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:53.916081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:53.916081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:53.916081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:53.916081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:53.916081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:53.916081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:53.916081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:53.916081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:53.916081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:53.916081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:53.916081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:53.916081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:53.916081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:53.916081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:53.916081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:53.916081 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:53.916481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:53.916481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:53.916481 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:57.683120 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:57.683589 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:57.683895 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:57.684428 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:57.684914 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.687048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:57.687048 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:57.687048 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:57.687048 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:57.687048 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:57.687048 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:57.687048 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.687048 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:57.687048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.687048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:57.687048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:57.687048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:57.687048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.687048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:57.687048 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:57.687388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.687388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:57.687388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:57.687388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:57.687388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.687388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:57.687388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:57.687388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:57.687388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:57.687388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:57.687388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:57.687388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:57.687388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:57.687388 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:57.687613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:57.687613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:57.687613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:57.687613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:57.687613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.687613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:57.687613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.687613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:57.687613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:57.687613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:57.687613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:57.687613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:57.687613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.687613 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:57.687835 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:57.688135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:57.688135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:57.688135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:57.688135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:57.688135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:57.688135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:57.688135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:57.688135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:57.688135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.688135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:57.688135 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:57.688845 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:57.688845 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:57.688845 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:52:57.689242 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:57.689689 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.690347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:57.690347 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:57.690347 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:57.690347 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:57.690347 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:57.690347 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:57.690347 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.690347 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:57.692310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.692310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:57.692310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:57.692310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:57.692310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.692310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:57.692310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:57.692310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:57.692310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.692310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:57.692310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:57.692310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:57.692310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:57.692310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:57.692310 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:57.692633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:57.692633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:57.692633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:57.692633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:57.692633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:57.692633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:57.692633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:57.692633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.692633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:57.692633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.692633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:57.692633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:57.692633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:57.692633 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.692843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:57.692843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:57.692843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:52:57.692843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:57.692843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:57.692843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.692843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:57.692843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:57.692843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:57.692843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.692843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:57.692843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:57.692843 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:57.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:57.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:57.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:57.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:57.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:57.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:57.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:57.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:57.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:57.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:57.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.693116 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:57.693327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.693327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:57.693327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:57.693327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:57.693327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:57.693327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:57.693327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.693327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:57.693327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:57.693327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.693327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:57.693327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:57.693327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.693327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:57.693327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:57.693327 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.693575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:52:57.693575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:52:57.693575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:52:57.693575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:57.693575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:57.693575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.693575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:57.693575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:57.693575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:57.693575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.693575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:57.693575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:57.693575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:57.693575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:57.693575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:57.693575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:57.693575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:57.693575 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:57.693846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:57.693846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:57.693846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:57.693846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:57.693846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:57.693846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.693846 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:57.693846 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:57.694522 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:57.694522 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:57.694522 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:57.694977 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:57.695506 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.698312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:57.698312 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:57.698312 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:57.698312 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:57.698312 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:57.698312 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:57.698312 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.698312 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:57.698312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.698312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:52:57.698312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:57.698312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:57.698312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.698312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:57.698312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:57.698312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:52:57.698312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.698312 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:52:57.698683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:57.698683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:52:57.698683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:57.698683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:52:57.698683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:52:57.698683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:57.698683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:52:57.698683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:57.698683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:52:57.698683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:52:57.698683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:57.698683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:52:57.698683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.698683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:52:57.698683 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.698916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:57.698916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:57.698916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:57.698916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.698916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:57.698916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:57.698916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:57.698916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:57.698916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.698916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:57.698916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:57.698916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:57.698916 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.699146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:57.699146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:57.699146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:57.699146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:57.699146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:57.699146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:57.699146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:57.699146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:57.699146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:57.699146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:57.699146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:57.699146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:57.699146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:57.699357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.699357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:57.699357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.699357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:57.699357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:57.699357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:57.699357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:57.699357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:57.699357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.699357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:57.699357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:57.699357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.699357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:57.699357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:57.699357 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.699692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:57.699692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:57.699692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.699692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:52:57.699692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:52:57.699692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:52:57.699692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:52:57.699692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.699692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:57.699692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:57.699692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:57.699692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.699692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:57.699692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:57.699692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:57.699692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:57.699692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:57.699692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:57.699692 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:57.699965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:57.699965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:57.699965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:57.699965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:57.699965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:57.699965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:57.699965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.699965 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:57.699965 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:57.700693 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:57.700693 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:57.700693 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:57.701724 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:57.701724 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:57.701724 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.701872 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:57.701872 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:57.701872 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:57.701872 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:57.701872 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:57.701872 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:57.701872 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.701872 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:57.703297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.703297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:57.703297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:57.703297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:57.703297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.703297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:57.703297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:57.703297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.703297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:57.703297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:57.703297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:57.703297 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.704036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:57.704036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:57.704036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:57.704036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:57.704036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:57.704036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:57.704036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:57.704036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:57.704036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:57.704036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:57.704036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:57.704036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:57.704036 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:57.704680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.704680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:57.704680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.704680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:57.704680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:57.704680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:57.704680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:57.704680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:57.704680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.704680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:57.704680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:57.704680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.704680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:57.704680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:57.704680 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.705387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:57.705387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:57.705387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:57.705387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:52:57.705387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:57.705387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.705387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:57.705387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:57.705387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:57.705387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:57.705387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:57.705387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:57.705387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:57.705387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:57.705387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:57.705387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:57.705387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:57.705387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:57.705387 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:57.706431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:57.706431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:57.706431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:57.706431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:57.706431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:57.706431 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:52:58.885594 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:52:58.886096 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:52:58.886096 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:52:58.886779 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:52:58.913771 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:52:58.913979 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:58.914461 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:52:58.914461 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:58.914461 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:52:58.914461 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:52:58.914461 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:58.914461 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:52:58.914461 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:58.914461 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:52:58.915111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:58.915111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:52:58.915111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:58.915111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:52:58.915111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:58.915111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:58.915111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:58.915111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:58.915111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:58.915111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:58.915111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:52:58.915111 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:58.915366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:52:58.915366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:58.915366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:52:58.915366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:58.915366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:52:58.915366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:52:58.915366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:58.915366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:52:58.915366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:58.915366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:52:58.915366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:52:58.915366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:58.915366 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:52:58.915639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:58.915639 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:52:58.915826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:58.915826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:52:58.915826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:52:58.915826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:52:58.915826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:52:58.915826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:52:58.915826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:58.915826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:52:58.915826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:52:58.915826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:58.915826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:52:58.915826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:52:58.915826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:52:58.915826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:52:58.915826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:52:58.915826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:52:58.916217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:52:58.916217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:52:58.916217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:58.916217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:58.916217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:52:58.916217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:52:58.916217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:52:58.916217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:52:58.916217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:52:58.916217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:52:58.916217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:52:58.916217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:52:58.916217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:52:58.916217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:52:58.916217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:52:58.916217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:52:58.916217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:52:58.916217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:52:58.916217 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:52:58.916557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:52:58.916557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:52:58.916557 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:53:02.683539 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:53:02.684140 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:53:02.684140 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:53:02.684602 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e46ca664-b5ab-4467-a6fb-c30711dc9a88, routes=[], if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:53:02.684805 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.685347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:53:02.685347 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:53:02.685347 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:53:02.685347 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:53:02.685347 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:53:02.685347 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:53:02.685347 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.685347 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:53:02.686365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.686365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:53:02.686365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:53:02.686365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:53:02.686365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.686365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:53:02.686365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:53:02.686365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.686365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:53:02.686365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:53:02.686365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:53:02.686365 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.686774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:53:02.686774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:53:02.686774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:53:02.686774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:53:02.686774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:53:02.686774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:53:02.686774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:53:02.686774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:53:02.686774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:53:02.686774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:53:02.686774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:53:02.686774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:53:02.686774 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:53:02.687125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.687125 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:53:02.687462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.687462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:53:02.687462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:53:02.687462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:53:02.687462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:53:02.687462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:53:02.687462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.687462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:53:02.687462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:53:02.687462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.687462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:53:02.687462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:53:02.687462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.687462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:53:02.687462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:53:02.687462 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.688480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 14:53:02.688480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:53:02.688480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.688480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:53:02.688480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:53:02.688480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:53:02.688480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.688480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:53:02.688480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:53:02.688480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:53:02.688480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:53:02.688480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:53:02.688480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:53:02.688480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:53:02.688480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:53:02.688480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:53:02.688480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:53:02.688480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:53:02.688480 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:53:02.689939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:53:02.689939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.689939 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:53:02.689939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:53:02.689939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:53:02.689939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:53:02.689939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89274) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 14:53:02.689939 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:53:02.690216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.690216 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:53:02.690216 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:53:02.690216 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:53:02.690216 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:53:02.690216 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:53:02.690216 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:53:02.690216 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.690216 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:53:02.690931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.690931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:53:02.690931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:53:02.690931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:53:02.690931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.690931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:53:02.690931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:53:02.690931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:53:02.690931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.690931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:53:02.690931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:53:02.690931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:53:02.690931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:53:02.690931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:53:02.690931 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:53:02.691403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:53:02.691403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:53:02.691403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:53:02.691403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:53:02.691403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:53:02.691403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:53:02.691403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:53:02.691403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.691403 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:53:02.691897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.691897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:53:02.691897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:53:02.691897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:53:02.691897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.691897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:53:02.691897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:53:02.691897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 14:53:02.691897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:53:02.691897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:53:02.691897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.691897 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:53:02.692601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:53:02.692601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:53:02.692601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.692601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:53:02.692601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:53:02.692601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:53:02.692601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:53:02.692601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:53:02.692601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:53:02.692601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:53:02.692601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:53:02.692601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:53:02.692601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:53:02.692601 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:53:02.693232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:53:02.693232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:53:02.693232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.693232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:53:02.693232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.693232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:53:02.693232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:53:02.693232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:53:02.693232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:53:02.693232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:53:02.693232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.693232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:53:02.693232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:53:02.693232 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.693723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:53:02.693723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:53:02.693723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.693723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:53:02.693723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:53:02.693723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.693723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 14:53:02.693723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 14:53:02.693723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^ Jun 08 14:53:02.693723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:53:02.693723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:53:02.693723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.693723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:53:02.693723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:53:02.693723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:53:02.693723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.693723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:53:02.693723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:53:02.693723 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:53:02.694146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:53:02.694146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:53:02.694146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:53:02.694146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:53:02.694146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:53:02.694146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:53:02.694146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:53:02.694146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:53:02.694146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:53:02.694146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:53:02.694146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.694146 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:53:02.694439 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:53:02.694481 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:53:02.694925 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:53:02.694925 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:53:02.695050 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:53:02.695347 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.695898 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:53:02.695898 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:53:02.695898 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:53:02.695898 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:53:02.695898 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:53:02.695898 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:53:02.695898 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.695898 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:53:02.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 14:53:02.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:53:02.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:53:02.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:53:02.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:53:02.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 14:53:02.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 14:53:02.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:53:02.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 14:53:02.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:53:02.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 14:53:02.696477 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 14:53:02.697135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:53:02.697135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 14:53:02.697135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:53:02.697135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 14:53:02.697135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 14:53:02.697135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:53:02.697135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 14:53:02.697135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.697135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 14:53:02.697135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.697135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:53:02.697135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:53:02.697135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:53:02.697135 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.697743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:53:02.697743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:53:02.697743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:53:02.697743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:53:02.697743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.697743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:53:02.697743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:53:02.697743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:53:02.697743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.697743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:53:02.697743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:53:02.697743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:53:02.697743 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:53:02.698032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:53:02.698032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:53:02.698032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:53:02.698032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:53:02.698032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:53:02.698032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:53:02.698032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:53:02.698032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:53:02.698032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:53:02.698032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.698032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:53:02.698032 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.698330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:53:02.698330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:53:02.698330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:53:02.698330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:53:02.698330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:53:02.698330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.698330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:53:02.698330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:53:02.698330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.698330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:53:02.698330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:53:02.698330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.698330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:53:02.698330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:53:02.698330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.698330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 14:53:02.698330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 14:53:02.698330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 14:53:02.698330 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.698757 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:53:02.700477 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:53:02.700666 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:53:02.700755 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:53:02.700872 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:53:02.701147 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:53:02.701386 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:53:02.701484 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.701844 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:53:02.701844 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:53:02.701844 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:53:02.701844 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:53:02.701844 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:53:02.701844 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:53:02.701844 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.701844 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:53:02.702467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.702467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:53:02.702467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:53:02.702467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:53:02.702467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.702467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:53:02.702467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:53:02.702467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.702467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:53:02.702467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:53:02.702467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:53:02.702467 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.702826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:53:02.702826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:53:02.702826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:53:02.702826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:53:02.702826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:53:02.702826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:53:02.702826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:53:02.702826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:53:02.702826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:53:02.702826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:53:02.702826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:53:02.702826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:53:02.702826 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:53:02.703132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.703132 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:53:02.703198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.703198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:53:02.703198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:53:02.703198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:53:02.703198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:53:02.703198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:53:02.703198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.703198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:53:02.703198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:53:02.703198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.703198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:53:02.703198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:53:02.703198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.703198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:53:02.703198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:53:02.703198 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:53:02.703663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 14:53:02.703663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:53:02.703663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.703663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:53:02.703663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:53:02.703663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:53:02.703663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:02.703663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:53:02.703663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:53:02.703663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:53:02.703663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:53:02.703663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:53:02.703663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:53:02.703663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:53:02.703663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:53:02.703663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:53:02.703663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:53:02.703663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:53:02.703663 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:53:02.704107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:53:02.704107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:02.704107 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Jun 08 14:53:03.886043 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89274) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 14:53:03.886756 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 14:53:03.886756 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89274) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 14:53:03.887198 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89274) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 14:53:03.917671 np0000002315 neutron-ovn-maintenance-worker[89274]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89274) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 14:53:03.917671 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:03.917911 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Traceback (most recent call last): Jun 08 14:53:03.917911 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:53:03.917911 np0000002315 neutron-ovn-maintenance-worker[89274]: txn.results.put(txn.do_commit()) Jun 08 14:53:03.917911 np0000002315 neutron-ovn-maintenance-worker[89274]: ^^^^^^^^^^^^^^^ Jun 08 14:53:03.917911 np0000002315 neutron-ovn-maintenance-worker[89274]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:53:03.917911 np0000002315 neutron-ovn-maintenance-worker[89274]: raise RuntimeError(msg) Jun 08 14:53:03.917911 np0000002315 neutron-ovn-maintenance-worker[89274]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:03.917911 np0000002315 neutron-ovn-maintenance-worker[89274]: Jun 08 14:53:03.918694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:03.918694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 14:53:03.918694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:53:03.918694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 14:53:03.918694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 14:53:03.918694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:53:03.918694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:53:03.918694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:03.918694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:53:03.918694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:53:03.918694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 14:53:03.918694 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:03.919046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 14:53:03.919046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:53:03.919046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 14:53:03.919046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:53:03.919046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 14:53:03.919046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 14:53:03.919046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:53:03.919046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 14:53:03.919046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:53:03.919046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 14:53:03.919046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 14:53:03.919046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:53:03.919046 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 14:53:03.919399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:03.919399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 14:53:03.919399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics [None req-c1466d3f-9469-479e-b642-5f8e90281650 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:03.919399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 14:53:03.919399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 14:53:03.919399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics work() Jun 08 14:53:03.919399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 14:53:03.919399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 14:53:03.919399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:53:03.919399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 14:53:03.919399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 14:53:03.919399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:53:03.919399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 14:53:03.919399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 14:53:03.919399 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 14:53:03.919801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 14:53:03.919801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 14:53:03.919801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 14:53:03.919801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 14:53:03.919801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 14:53:03.919801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:03.919801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:53:03.919801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 14:53:03.919801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 14:53:03.919801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 14:53:03.919801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics next(self.gen) Jun 08 14:53:03.919801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 14:53:03.919801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics with self.create_transaction( Jun 08 14:53:03.919801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 14:53:03.919801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics self.result = self.commit() Jun 08 14:53:03.919801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 14:53:03.919801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 14:53:03.919801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise result.ex Jun 08 14:53:03.919801 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 14:53:03.920292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 14:53:03.920292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 14:53:03.920292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 14:53:03.920292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 14:53:03.920292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 14:53:03.920292 np0000002315 neutron-ovn-maintenance-worker[89274]: ERROR futurist.periodics